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ABSTRACT 



A study examined major performance standards issues 



in the context of employment and training programs targeted toward 
the Aid to Families with Dependent Children (AFDC) population. 
Telephone interviews were conducted with representatives from Job 
Training Partnership Act (JTPA) programs and departments of public 
welfare in nine states. A significant amount of state-level activity 
was aimed at improving the level and quality of services available to 
AFDC recipients^ including the hard-to-employ. All of the programs 
identified placed a premium on interagency coordination and on 
flexible and enriched program design. Welfare agencyt respondents 
considered limited funds and the lack of an adequate consensus as to 
the proper role of employment and training programs and their 
relationship to public welfare programs to be the greatest barriers 
to providing services to AFDC recipients. Among the JTPA respondents, 
performance standards were the most commonly cited barrier. Six of 
the nine states studied had undertaken efforts to develop local 
performance standards beyond those produced by tf:e Department of 
Labor's adjustment model. The message regarding performance standards 
was mixed. Although they were considered the biggest obstacle to 
providing services, they were also seen as one of the most effective 
means of improving the level and quality of services. CMN) 
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EXECUTIVE SUMMARY 



Performance standards have long been regarded as an important manage- 
ment tool for employment and training programs. They provide basic information for 
addressing key strategic planning and operational decisions, and allow funding agen- 
cies to determine the extent to which programs are moving in a direction consistent 
with their long-term goals. For the manager who operates within a dynamic, rapidly 
changing environment, performance standards provide a timely and useful indication 
of program effectiveness. 

The art of managing with performance standards has been developed largely 
on the basis of experience gained under enr.ployment and training programs author- 
ized first by the Comprehensive Employment and Training Act (CETA) and currently 
oy the Job Training Partnership Act (3TPA). While employment and training pro- 
grams authorized by other sources have also relied upon performance standards, 
those administered by the Department of Labor have assumed a leadership position in 
this area of program management. 

In drafting JTPA, both the Congress and the Administration recognized the 
importance of program outcomes, and established a performance standards system as 
the cornerstone of the new legislation. In many respects, 3TPA was designed as a 
performance-driven program, intended to clearly link program inputs with program 
outputs. To achieve this, an elaborate management structure was designed to set 
numeric performance scores, and provide rewards and sanctions for good and poor 
performance standards, respectively. 

Although 3TPA*s performance standards system is expected to (ulfill a 
critical management role, it can also create "perverse incentives" to serve the most 
job-ready applicants in short-term, l^w-cost programs. To address these potential 
effects, 3TPA contains at least five mechanisms, including: 

1. A requirement that 90 percent of all enrollees be disadvantaged; 

2. Targeting provisions requiring specified levels of service to youth, high 
school dropouts, and welfare recipients; 

3. Authorization for Governors to adjust Service Delivery Area (SDA) 
standards to account for variations in participant characteristics and 
other factors affecting performance; 

The establishment of performance standards at a minimally acceptable 
level; and 

5. The availability of financial incentives that Governors can use to 
influence SDA behavior. 
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Respite these efforts, there has been growing concern that performance 
standards have reduced services to the hard-to-employ, and limited the level of 
investment made in participants. Indeed, this has been argued by practitioners, 
researchers, and representatives from the Congress, Many believe that the perfor- 
mance standards system has shifted the attention of programs away from long-term 
employability development to the creation and management of a low-cost, high- 
volume service delivery system. These concerns have been particularly directed at 
the Aid to Families with Dependent Children (AFDC) population and especially those 
identified as hard-to-employ. Despite the presence of specific targeting provisions, 
many believe that service providers "cream" from among the eligible population of 
welfare recipients and provide them with limited, short-term services. 

The seriousness of these concerns has caused the Congress and the Depart- 
ment of Labor to reassess the performance standards system critically and consider 
modifications to it. At the same time, the Congress and Administration are also 
considering amendments to JTPA that would encourage services to AFDC recipients 
and other hard-to-employ groups, as well as legislation that would restructure the 
welfare system and the manner in which training services are provided to public 
assistance recipients. Since several of the legislative proposals suggest the creation 
of performance standards to guide the provision of services to welfare recipients, an 
assessment of such standards is quite important and can provide timely input to 
current policy deliberations. 

This study is designed to assess a series of performance standards issues in 
the context of employment and training programs targeted on the AFDC population, 
and especially those who are hard to serve. The study has three major objectives, 
including: 

1. To describe and evaluate the effects of JTPA performance standards 
on reducing welfare costs, with specific reference to who is served, the 
services that are delivered, and the effectiveness of programs for 
welfare recipients, 

2. To describe and evaluate the effects of performance standards on 
service providers in other employment and training programs for indi- 
viduals similar to those who uill be served by the proposed legislative 
initiatives, 

3. To suggest a set of {\ierformance measures and a methodology to estab- 
lish standards for targeted welfare programs, such as those contained 
in the legislative proposals. 

In this report, we present the initial results obtained from telephone inter- 
views conducted with representatives from JTPA and departments of public welfare 
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in nine states that account for approximately *0 percent of the total U.S. AFDC 
caseload. The objectives of these interviews were to describe and document: (1) 
policies and procedures that encourage services to the hard-to-employ; (2) existing 
welfare performance standards and adjustment procedures; (3) key concerns with 
performance standards management; and (^) major constraints in serving AFDC 
recipients, including the hard-to-employ. 

Key Findings Obtained from Telephone Interviews with State Officials 

Overall, our findings revealed a significant amount of state-level activity 
aimed at improving the level and quality of services available to AFDC recipients, 
including those believed to be hard-to-employ. Taking advantage of the flexibility 
provided by the federal government to operate a variety of work-welfare programs, 
nearly every state relied upon gubernatorial support to leverage at least one new 
program initiative reflecting a particular policy perspective and prescription for 
strengthening the relationship between work and welfare. These efforts were 
planned and implemented in the context of special demonstration programs or large- 
scale welfare reform initiatives. The size and scope of these initiatives were deter- 
mined as much by funding availability as by the degree of state certainty over how to 
best implement a program that would be more responsive to the needs of the welfare 
population. 

As would be expected, the states we interviewed were in different stages of 
developing and implementing their initiatives. In a few instances, these initiatives 
were several years old, state-wide in scope, and had become institutionalized. In 
other cases, they were in the initial implementation stages, while in a few states 
program and policy development were still on the drawing board. However, acn.ss 
all states several key themes emerged. The first was the highly targeted nature of 
the program initiative. In each instance, states had identified subgroups of the 
AFDC (and other) populations which they believed were hard-to-employ. These 
subgroups, however, varied substantially, and ranged from teen parents, to long-term 
recipients, to those with limited education. 

Second, all of the programs placed a premium, both conceptually and pro- 
grcimmatically, on interagency coordination, stressing the development of working 
relationships between JTPA, departments of public welfare and social services, and 
education. This emphasis reflected the common perception that no one program 
alone could address the multiple needs of the AFDC recipient. JTPA, for example, 
provides training and employment services, but has limited resources available for 



supportive services and remedial education, while departments of public welfare 
have support service resources but are more constrained with respect to training and 
education. By coordinating the resources of both programs it was believed that the 
quality of service delivery could be substantially improved. 

Above and beyond coordination, flexible and enriched program designs also 
characterized most of the demonstrations. While a few states had designed their 
initiatives to follow very prescribed service patterns, most adhered to the notion 
that services had to be matched to the needs of individuals. Thus, service providers 
were given substantial latitude in designing their specific programs and were encour- 
aged to adopt strategies that maximized a flexible approach to meeting client needs. 

To a large extent, the features of state initiatives reflected an effort to 
address many of the obstacles respondents believed inhibited services to AFDC 
recipients, including the hard-to«employ. These obstacles were identified during our 
discussion of state initiatives and after asking respondents about their perceptions of 
key factors limiting the quality and level of services to AFDC recipients, A key 
objective in asking this question was to determine the role of performance standards 
in shaping targeting and program mix decisions. 

Overall, o^ir results indicate that while there is a need to improve certain 
aspects of 3TPA's performance standards system, such improvements were not 
consistently viewed as among the most important to enhancing the level and quality 
of services for the welfare population. This was particularly true among welfare 
agency respondents, who most frequently mentioned limited funds and the need to 
develop a better consensus on the proper role of employment and training programs 
and their relationship to public welfare programs. This result emerged despite the 
fact that in seven of the nine states we interviewed, welfare departments had 
designed and implemented their own version of performance standards. 

Among our JTPA respondents, performance standards were mentioned more 
frequently; five of the nine respondents cited them as a key factor inhibiting services 
to AFDC recipients and particularly the hard-to-employ. For the most part, 3TPA 
respondents saw performance standards as creating too many risks to enroll hard-to- 
employ individuals. The types of risks mentioned most often included loss of six 
p<>rcent funds and "looking bad" in the public's eye and particularly to local Private 
Industry Councils, However, in contrast to the expectation that the overwhelming 
majority would view performance standards as a key constraining factor, only 55 
percent responded in this fashion. 



Since states have the option to negotiate local performance beyond the 
standards produced by the Department of Labor's adjustment model, we asked them 
if they had engaged in efforts to do so. Of the nine states, six responded that they 
had developed such policies. However, these same states indicated that only infre- 
quently did SDAs request such additional adjustments. This was attributed to the 
complexity of the JTPA performance standards system, including state adjustment 
policies, as well as to the fact that most SDAs had little difficulty meeting their 
performance expectations. Absent a major national or state policy emphasis on 
servir;g hard-to-employ individuals, SDAs had little need to request ac ' al 
adjustments to their performance standards. 

The mixed message we obser^'ed regarding the effects of performance 
standards was further reinforced when we asked respondents to identify the most 
effective means for improving the level and quality of services to AFDC recipients. 
Across all states and respondents, improved coordination, augmenting the availability 
of support services, and increased funding were cited most frequently, followed by 
improvement in the management of performance standards. Apparently, while 
several respondents viewed standards as constraining the process of decisionmaking, 
they did not believe that they were among the most important steps that could be 
taken to improve the responsiveness of employment and training programs to AFDC 
recipients. 

In part, these findings should not be surprising since the issues raised by the 
respondents relate to fundamental needs of the target population and major institu- 
tional barriers faced when developing a flexible and responsive set of program ser- 
vices. However, the findings also reflect the opinions and perspectives of individuals 
who are removed from the daily problems faced by line managers. Therefore, one 
would expect these individuals to be more focused on broad policy questions rather 
than on specific service delivery concerns* It still appears, however, that while 
modifications to the performance standards system may be necessary, they may not 
be a sufficient condition for improving service delivery to the welfare population. 




1 -0 INTRODUCTION TO THE STUDY 



1.1 Objectives of the Study 

This report is the first of three to be produced as part of a study investigat- 
ing the role and use of performance standards for employment and training programs 
serving welfare recipients. The study has three major research objectives, including: 

!• To describe and evaluate the effects of 3TPA performance standards 
on reducing welfare costs, with specific reference to who is served, the 
services that are delivered, and the effectiveness of programs for 
welfare recipients. 

2. To describe and evaluate the effects of performance standards on 
service providers in other employment and training programs for indi- 
viduals similar to those who will be served by the proposed legislative 
initiatives. 

3. To suggest a set of performance measures and a methodology to estab- 
lish standards for targeted welfare programs, such as those contained 
in the legislative proposals. 

Our approach to these objectives consists of three tasks. The first task is 
to assess the influence of performance standards on targeting, program mix, and 
investment decisions. Here, we conduct a qualitative analysis of the role of perfor- 
mance standards in planning and providing services to AFDC recipients, and particu- 
larly those that are hard-to-employ. 

The analysis is based upon telephone interviews with the nine states that 
account for the largest share of AFDC recipients in the U.S. and field visits to a sub- 
set of three states. In each state, interviews are conducted with representatives 
from both 3TPA and welfare departments, while during the field visits we also visit 
the largest urban SDA as well. The states and SDAs included in the study are: 



State Telephone Field 

Included in Study Interview Visit SDA 



Florida x x Miami 

Illinois X 

Michigan x 

New Jersey x 

New York x x New York City 

Massachusetts x x Boston 

Ohio X 

Pennsylvania x 

Texas x 



The telephone survey is designed to provide descriptive information regard- 
ing welfare performance standards and provision of to hard-to-employ individuals. In 
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each state, two interviews are conducted: one with the manager of JTPA perfor- 
mance standards, and another with the individual responsible for employment and 
training in the state welfare department. These two respondents are asked a series 
of brief questions designed to describe and document: I) policies and procedures that 
encourage services to hard-to-employ individuals; (2) existing welfare performance 
standards and adjustment procedures; (3) key concerns with performance standards 
management, with special reference to the mix, level, and setting of standards; and 
major constraints in serving the hard-to-employ. To the extent available, respon- 
dents are also asked to mail relevant documents, policies, and performance related 
data to support our effort. 

The information obtained from our telephone survey is used for two pur- 
poses. First, we summarize key findings for each individual state, highlighting 
differences between JTPA and non-JTPA programs. Second, we conduct a cross-site 
analysis on a topic-by-topic basis in order to portray how states have addressed key 
issues related to welfare performance standards and the delivery of services to the 
hard-to-employ. 

The SDA site visits are intended to illuminate the interaction of perfor- 
mance standards with key service delivery policies and procedures. The primary 
areas of inquiry guiding the field visits, include: (I) policies and practices used to 
shape recruitment and selection, (2) criteria for selecting a program mix and match- 
ing individual needs with services, and (3) constraints imposed by performance stan- 
dards. 

To ground our qualitative examination of performance standards in an 
analytic framework, we start with the proposition that while there is widespread 
variation in what drives programs to behave as they do, programs do share a common 
set of constraints. Understanding these constraints is key to understanding program 
behavior and the role of ^(Serformance standards. 

For example, some programs are highly "risk-averse"; they conservatively 
follow plans without "rocking the boat" unnecessarily. Others, perhaps motivated by 
the desire to best serve their clientele, are willing to experiment, willing to drive 
themselves to greater levels of performance. 

But all programs are funded with public funds, operating within budgets set 
by higher levels of government. And each program operates within a local environ- 
ment that determines the demographic mix of potential enrollees and the overall 
need for services. Programs also operate within a particular political context; 
community advocates and the private sector may be more or less active, the sur- 
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rounding media and community may be more or less supportive, or local contractors 
may be more or less powerful. 

Furthermore, the environment in which programs operate is one ?n which 
"hard" facts about the effects of various programs activities on various client groups 
are difficult to find. Should intensive, costly efforts be directed at reducing depen- 
dency on public assistance for a relatively jmall number of long-term welfare recipi- 
ents, or should funds be allocated to extensive, but short-term, efforts to serve large 
numbers of people? in the absence of state or national regulations, we expect pro- 
grams to develop a wide variety of services with an equally wide variety of out- 
comes, all based on local experience and conditions. 

We view performa iCe standards as another constraint. They are intended to 

direct programs toward particular targets (e.g., percent of entered employment) 

which are judged to be important in achieving longer-term goals. The performance 

targets provide each program with a yardstick against which to measure its own 

performance. Those performing below their expected levels may be encouraged to 

reassess their behavior. If the program feels that it is already acting optimally, it 

« 

may request adjustments to the standards that it faces. Otherwise, it will be 
encouraged to change its behavior. 

But it is important to note that all of the constraints facing a program only 
affect behavior to the extent that they are binding on the constraint agent. For 
example, performance standards might be set so leniently that no change in program 
behavior would be required to meet them. 

More important, the performance standard constraints might be ineffective 
because programs cannot adjust "heir behavior to meet them. The standards might, 
for example, provide very strong incentives to serve certain subgroups, but if pro- 
grams cannot recruit such applicants, they will not be able to respond to the perfor- 
mance standard system. In that case, the recruitment constraint is operative before 
the performance standard constraint. As another example, suppose a program has 
had a long, historically productive association with a particular vendor. Even a 
strong incentive to change service mix niay go unheeded if it implies breaking up the 
established pattern of "doing business." 

Developing an understanding of the effects of performance standards 
requires identifying where in the program planning and delivery process they can 
become operational constraints. Exhibit 1.1 presents the broad conceptual frame- 
work that links together the outcomes of interest in this study with the key factors 
that can affect them. As can be seen, the model largely resembles the rational 
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Exhibit 1.1 

Framework for Analyzing the Effects of 3TPA Performance Standards on Clients, Services and Cc^ts 



FACTORS INFLUENCING SERVICE PROVIDER DECISIONS 



FEDERAL 




Performance Standard 




Pol Icies 




• Choice of mt/dsures 




• Level of standards 




• Adjustment provisions 




Other Pol icies 




• Reporting requirements 




* Technical assistance 




information 





STATE POLICIES 

• Nonfederal measures 
and standards 

• Adjustment methodology 

• Incentives, technical 
assistance and sanctions 

• Reporting requirements 

• Concern over the hard- 
to-employ 



PR06RAM/SDA 
CHARACTERISTICS 

• Degree of PIC involve- 
ment 

• Concern with performance 
standards 

• Concern with coordina- 
tion 

• Pol icies regarding 
hard-to-serve 

• Management quality 





ENVIRONMENTAL 




CHARACTERISTICS 




• Population character- 




istics 




• Labor market charac- 




teristics 




• Geographic character- 




istics 



SERVICE PROVIDEf. DESIGN DECISIONS 



TARGETING, RECRUITMENT, 
AND ENROLLMENT 



PLANNED PROGRAM MIX 



SERVICE PROVIDER ARRANGEMENTS 



SERVICE DELIVFRY 



CI ient Mix 



Program Services 



Program Costs 
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planning process. Service provider decisions regarding who to serve, in what pro- 
grams, and through what contract means have a direct bearing on enrollment, service 
delivery, and costs. These decisions, in turn, are influenced by other conditioning 
factors imposed on service deliverers by the environment, higher levels of govern- 
ment, and their own organizational characteristics. 

Within this framework, performance standards can be seen to have an 
important influence on the types of welfare recipients enrolled, the services offered 
to them, and the associated cost. Our approach to the qualitative analysis relies on 
this framework to identify how performance standards may influence service pro- 
vider decisions. 

The second task is to conduct a series of validation tests to assess the 
predictive validity of alternative performance measures using experimentally derived 
impact estimates obtained from the AFDC Homemaker-Home Health Aide Demon- 
strations. These demonstrations provided training and subsidized employment as 
homemaker-home health aides to AFDC recipients, who then supplied in-home ser- 
vices to functionally impaired clients. Thus, the demonstrations sought to alleviate a 
major social problem, that being welfare dependency. The Health Care Financing 
Administration (HCFA) was the lead agency for implementing the demonstrations in 
seven states: Arkansas, Kentucky, New Jersey, New York, Ohio, South Carolina, and 
Texa:J. Operations began in January 1983 and ended June 30, 1986. Participation in 
both the training and service components of the demonstrations was voluntary. To 
permit rigorous evaluation of demonstration effects, eligible applicants (both poten- 
tial trainees and potential clients) were randomly assigned in equal numbers to 
treatment and control groups. 

At the time of enrollment, both groups were administered a baseline ques- 
tionnaire to identify their prior welfare and employment histories as well as their 
persona' characteristics. Following baseline, treatment and control group members 
were subject to one foUowup interview, administered between 5 and 32 months later? 
on average, the followup period was 12 months long. The followup data, combined 
with welfare payment information obtained from state administrative records, pro- 
vided the basis for estimating (experimental) impacts on welfare dependency, 
employment, and earnings. Thus, these data offer a unique opportunity to validate 
welfare performance measures with experimentally-based impact estimates. 

In selecting these data to conduct the validation tests, we realized that the 
demonstrations represent one specific approach to training and placing welfare 
recipients. While it might be preferable to perform the tests on a broad range of 



employment and training programs for welfare recipients, such data are not avail- 
able. Thus, we view the AFDC Homemaker-Home Health Aide Demonstrations as 
the best choice for four key reasons, including: 

1. The data focus on the specific population of interest, namely AFDC 
recipients. On average trainees served in the program had not worked 
for 3^ month:: prior to entry, were 31 years of age, cared for two 
dependent children, were black or Hispanic (72 percent), and had not 
gone beyond high school (78 percent). In many respects, they resemble 
the target group of interest to this study. 

2. The data permit estimation of experimentally-based welfare reduction 
impacts. 

3. The data contain detailed information on prior work history and wel- 
fare status. 

^. The findings can be combined with those obtained previously and with 
the research results produced by MDRC to yield a comparative frame- 
work for assessing the validity of alternative performance measures. 

Using these data, we construct two validation benchmarks— impacts on 
welfare reductions and earnings— and test the validity of a number of alternative 
measures that vary by both definition and measurement. The measures we test focus 
on: 1) employment status; 2) earnings; 3) return to school/enrollment in training 
status; and ^) welfare status. Each measure is constructed at the time of program 
termination as well as over a three- and six-month post-program period. 

To judge the relative merits of alternative measures, we rely on three 
criteria. The first is face validity; that is, the measures will have to be related to 
the purpose of the program and be viewed as having intuitive appeal. The second 
criteria is feasibility. Here, our concern is with the practical difficulties associated 
with obtaining state and local data. The performance measures selected must not 
present service providers with undue costs or problems in collecting and reporting 
the necessary data. To address the last criterion— predictive validity— we rely on 
three tests. 

First, we examine simple correlations between each candidate measure and 
the validation benchmarks. The relative magnitude of the resulting correlations thus 
provide an indication of the relative validity of each measure. In practice, however, 
more than one measure may be used. As a result, we also conduct joint and incre- 
mental tests of validity. Alternate groups of indicators are assessed against each 
other, in an effort to determine by how much validity increases when selected mea- 
sures are added to another. 
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The success of a performance management system is very dependent on the 
framework adopted to adjust standards for differences in client characteristics and 
local economic conditions. Thus, our third task examines the types of adjustment 
models that would be most appropriate for targeted welfare training programs. 
Focusing on the most promising Hdate performance measures, we rely upon 
4"nultivariate statistical techniques to determine those potential adjustment factors 
that best explain variations in each indicator. 

1.2 Purpose and Organization of the Report 

This report presents the .findings obtained from 18 telephone interviews 
with state representatives from both JTPA and departments of public welfare. The 
report is designed to provide descriptive information on the current status of per- 
formance standards management, and obstacles to and strategies for providing 
quality services to the welfare population. 

Importantly, the findings presented in this report reflect the views and 
opinions of individuals at the state level who are responsible for the management of 
employment and trcining programs. Because these individuals are somewhat 
removed from the c'^ctual delivery of services, our findings do not fully reflect the 
issues, concerns, and perspectives of on-line staff with responsibilities for program 
delivery. The findings do, however, provide important insights regarding the prob- 
lems and issues which states have experienced, first in managing their employment 
and training programs and second, in introducing policies to support more effective 
cervice delivery to welfare recipients and the other hard-to-employ subgroups. 

Chapter 2 of the report discusses key issues in the development of perfor- 
mance standards. Chapter 3 presents a review of the current program and policy 
landscape surrounding programs serving welfare recipients. In Chapter we review 
the current status of performance management in both 3TPA and welfare programs, 
and then discuss respondent views of key factors influencing service delivery to the 
AFDC population. 
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KEY ISSUES IN DEVELOPING PERFORMANCE STANDARDS 



In this chapter, we discuss how performance standards have evolved in 
employment and training programs and highlight major issues that have arisen in 
their use. We then present a critical discussion of key challenges in designing a 
performance management system for programs serving AFDC recipients and other 
hard-to-employ groups. 

2.1 The Role of Performance Standards in Employment and Training Programs 

Performance standards have long been regarded as an important manage- 
ment tool for employment and training programs. They provide basic information for 
addressing key strategic planning and operational decisions, and allow funding agen- 
cies to determine the extent to which programs are moving in a direction consistent 
with their long-term goals. For the manager who operates within a dynamic, rapidly 
changing environment, performance standards provide a timely and useful indication 
of program effectiveness. 

The art of managing with performance standards has been developed largely 
on the basis of experience gained under employment and training programs author- 
ized first by the Comprehensive Employment and Training Act (CETA) and currently 
by the 3oh Training Partnership Act (JTPA). While employment and training pro- 
grams authorized by other sources have also relied upon performance standards, 
those administered by the Department of Labor have assumed a leadership position in 
this area of program management. 

The most concrete attempt to rely on a structured approach to performance 
management was introduced to the CETA program in 1977 under the auspices of the 
Grant Review Guidelines. These guidelines represented an effort by the Department 
of Labor to construct the best possible mechanism for assessing local program per- 
formance, given the limitations of federal reporting systems and the commitment 
not to engage in new or additional data collection activities. 

When first introduced, the Grant Review Guidelines relied upon three 
performance clusters and a comparative framework to set and assess performance 
achievement. These are presented in Exhibits 2.1 ar ' ^.2. As can be seen, there 
were 10 performance indicators separated into three c. .ers, including a termina- 
tion cluster, a cost cluster, and a fund utilization cluster. The use of clustering was 
initially adopted to account for the wide diversity of goals in the CETA program 
which, it was believed, could not adequately be measured by only one or even a few 
indicators. 
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Exhibit 2.1 

Performance Clusters and Indicators Used in the 
G rant Review Guidelines under CETA 

T ermination Cluster 

1) POSITIVE TERMINATION Ri\TE: 

2) ENTERED EMPLOYMENT RATE (A) : 

3) ENTERED EMPLOYMENT RATE (B) : 

4) NONDIRECT PLACEMENT RATE (A): 

5) NONDIRECT PLACEMENT RATE (B): 

Cost Cluster 

1) COST PER POSITIVE TERMINATION: 

2) COST PER ENTERED EMPLOYMENT: 

3) COST PER NONDIRECT PLACEMENT: 

Fund Utilization Cluster 

1) CARRYOUT RATE: 100| - (Funds Available - Accrued Expenditures ) 

Funds Avai table 

2) ADMINISTRATIVE COST RATE: Administrative Expenditures 

Total Accrued Expenditures 



BEST COPY AVAILABLE 
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Total Positive Terminations 
Total Terminations 

Entered Employment 
Total Terminations 

Entered Employment 
Total Positive Terminations 

Nondirect Placements 
Total Terminations 

Nondirect Placements 
Entered Employment 



Total Accrued Expenditures 
Positive Terminations 

Total Accrued Expenditures 
Entered Employment 

Total Accrued Expenditures 
Nondirect Placements 



Exhibit 2.2 

Illustration of Comparative Framework Used In the Performance Review Guidelines Under CETA 





COMPARATIVE DATA 


REGION 


NATION 


Terra»n6tion Cluster 


Planned (New 
(r 1 sea 1 Year) 


Actua 1 (Previous 
(Fiscal Year) 


Low 


33? 


67% 


High 


Low 


33? 


61% 


High 






































































































































xplain or justify if planned performance for 
he cluster is below previous performance or the 
owest regional third. 


Explanation 
Third 









Lo..jr performance standards may be due to: 

— Service to specific target groups 
Local labor market conditions 

— Issues related to program operations 



BEST COPV WMLKBLE 



' - J. 



Referring to the termination cluster, for example, it can be seen that there 
were five c^paraie performance indicators, representing an increasing amount of 
specificity with respect to program outcomes. The positive termination rate, for 
instance, captured the bn.adest level of program success by including both unsubsid- 
ized job placements and other types of positive outcomes such as returning to school 
or entering the military, (n contrast, the entered employment was a more refined 
outcome variable that provided a general measure of the degree to which the CETA 
program resulted in unsubsidized employment. By examining the remaining three 
indicators, it should be apparent that they became ev^en more specific by measuring 
the internal composition of both the positive terminations and entered employment 
rates. 

On an annual basis, local service providers, referred to as Prime Sponsors, 
would develop planned levels of performance achievement and negotiate them with 
regional offices of the Department of Labo". The basis for negotiation is reflected 
in Exhibit 1.2. As can be seen, in setting final performance standards, planned 
performance levels would be evaluated against those of the previous year as well as 
regional and national performance levels. While national performance scores would 
be used only for general reference, any proposed standard that fell below either the 
previous yearns achievement or the region^s lowest or highest thirds would require an 
explanation and negotiation. 

The actual use of this comparative method for setting performance stan- 
dards was based upon an examination o: oach of the i ^icators within the context of 
its cluster. On a case-by-case basis, Prime Sponsors were permitted to plan perfor- 
mance levels below what would otherwise be judged as low performance as long as 
the remaining indicators within a cluster we^e viewed as satisfactory. Similarly, if 
planned performance fell within the regional third requiring explanation, but repre- 
sented an improvement over the previous year, an explanation from the Prime Spon- 
sor was not necessarily required. 

When negotiating local performance ^^ndards, the Grant Review Guidelines 
required regional offices to explicitly consider local management capabilities as well 
as programmatic and environmental factors. These variables were referred to as 
"interpretive factors" and included measures of participant characteristics, program 
mix, and local environmental conditions. As was stated in a Department of Labor 
memorandum: 
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...It will be incumbent on the federal representative, who is knowledgeable 
concerning the environmental and programmatic aspects of the prime 
sponsor, to make judgements as to the factors and the strength of the 
factor considered to be operating to determine if they are adequate to 
explain planned performance. 

While the Grant Review guidelines represented a structured strategy for 
objectively setting and assessing local performance standards, they were subject to 
criticism on two main grounds. The first wai> that because the standards were lim- 
ited to termination-based measures, they did not adequately capture CETA*s long-run 
goal of easing barriers to labor force participation, securing unsubsidized employ- 
ment for participants, and increasing participants* earned incomes. While a priori 
^reasoning suggested that the existing measures were valid proxies for longer-term 
program success, it was believed, especially by the practitioner community, that 
foUowup indicators and those measuring changes in individual employment and earn- 
ings status would better reflect the goals of CETA. 

The second criticism was that the Grant Review Guidelines were ultimately 
too subjective, relying on the intuitive judgment of regional offices to take into 
consideration systematically the factors that affected local Prime Sponsor perfor- 
mance. Although most agreed that the interpretive factors used by the regional 
offices had merit, most did not agree that uniform and equitable application could be 
achieved on the basis of subjective judgment alone. 

Partly in response to these criticisms, the 1978 CETA Amendments included 
two new provisions designed to strengthen the performance management system. 
The first was that Prime Sponsors would be required to report to the Secretary of 
Labor information on participants* employment and earnings some time after leaving 
a program. Although it did not mandate uniform data collection and measurement, 
this requirement was viewed as necessary to improve local program management and 
strengthen the negotiation framework embodied in the Grant Review Guidelines. To 
support the legislative amendment, the Department of Labor drafted broad regula- 
tions requiring the collection of local followup data and their use in the local man- 
agement process. This was later supplemented with technical assistance and training 



U.S. Department of Labor, Employment and Training Administration, Field 
Memorandum No. 3*7-77, Washington, D.C., p. 18. 
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funded by the Department of Labor, although the methods used to conduct and use 
followup data were left to the discretion of Prime Sponsors.^ 

The second related amendment required the Secretary of Labor to establish 
local performance standards that accounted for differences in local conditions, 
including the nature and needs of the local population to be served and local labor 
market conditions. In response to this, the Department of Labor steadily moved 
toward the adoption of a quantitative modeling technique to setting local perfor- 
mance standards. 

In Fiscal Year 1982, quantitative modeling was formally introduced into the 
Grant Review Guidelines. Rather than relying on previous performance and that of 
the region for judging the overall adequacy of local plans, regression models were 
used to establish performance benchmarks for each Prime Sponsor. These models 
controlled for a variety of measurable factors that affected local performance and 
were intended to objectively neutralize key differences among Prime Sponsors with 
respect to client characteristics and economic conditions. Benchmarks -were 
expressed in terms of ranges to account for modeling imprecision and to introduce 
flexibility to the management process; they were also constrained to five of the 
original ten performance measures in an effort to better focus attention on those 
measures tha: were believed most important, including: (I) entered employment 
rate, (2) cost p<^r entered employment, (3) positive termination rate, W cost per 
positive termination, and (5) average wage at placement. 

Within this revised framework, the Department of Labor believed that 
performance standards management would De substantially improved, largely through 
greater objectivity in the goals-setting process. It was still recognized, however, 
that the model adjustments could not be expected to control for all relevant factors 



U.S. Department of Labor, Employment and Training Administration, 
Comprehensive Employment and Training Act Regulations , Final Rule and Proposed 
Rule, Washington, D.C., Section 676.22(dX2). 

See, for example, U.S. Department of Labor, Employment and Training 
Administration, CETA: Management Information System Program, Functional Man- 
agement Information System , Washington, D.C., July 1980. See p. 17 for followup 
recommendations. 

For a review of the performance review gr ines used prior to the 
modeling approach, see: Nancy Beckley, The Developm':>>>, ^nd Use of ETA's Short- 
Term Indicators , U.S. Department of Labor, Employment and Training 
Administration, February 1980. 
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affecting local performance and that negotiation between the regional office and 
Prime Sponsors would remain important. 

By and large, these revisions were viewed favorably by the practitioner 
community. Although the Department of Labor had to devote consideioOle time and 
effort to convincing program operators of the merit, intuitive appeal, and practical 
value of quantitative modeling, most accepted it and believed it was an improvement 
over the previous strategy. For the next two years, the Department of Labor contin- 
ued with its revised approach and devoted substantial resources to improving the 
accuracy of the adjustment models and the manner in which they were applied for 
management purposes. Thus, as CETA came to a close in September 1983, over six 
years of experience had been gained with an increasingly sophisticated approach to 
performance standards management. 

In many respects, the implementation of 3TPA in October 1983 continued 
the growing emphasis that was placed on performance management in employment 
and training programs. However, unlike CETA, 3TPA was crafted in the image of a 
performance-driven program with a much more elaborate management structure 
designed to clearly link program inputs and outputs. Moreover, the crafters of the 
legislation included very specific guidelines regarding the choice of performance 
measures, the basis upon which to develop numeric standards, and rewards and sanc- 
tions for exceeding and consistently falling below performance expectations. And, of 
equal importance was the transfer of responsibility for managing performance stan- 
dards from the Department of Labor's regional offices to state government. In many 
respects, the performance standards system created for JTPA has been character- 
ized as an offort to introduce market-like forces and incentives into federal employ- 
ment and training programs.^ 

Like CETA, the performance measures established under JTPA are intended 
to be reasonable proxies for long-tern net impacts, and when used in an objective 
manner, are expected to promote the Act's overall goals. Currently, JTPA operates 
with seven termination-based performance standards; three thirteen-week post- 
program measures have been identified as candidates for Program Year 1988. These 
measures and their national averages are presented below. 



See Thomas Bailey, "Market Forces and Private Sector Processes in Gov- 
ernment Policy: The Job Training Partnership Act, in Journal of Policy Analysis and 
Management , Vol. 7, No. 2, 1988, pp. 300-315. 
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3TPA Title IIA Performance Standards 



(PY 1986/87) 



Performance Measure National Average 

Adult entered employment rate (AEER) 62.00% 

Adult cost per entered employment (ACEE) $*37*.00 

Adult average wage at placement (AAWP) $^f.9l 

Adult welfare entered employment rate (AWEER) 51»00% 

Youth entered employment rate (YEER) . *3»00% 

Youth positive termination rate (YPTR) 75*00% 

Youth cost per positive termination (YCPT) $^f900.00 

Adult post-program employment rate N/A 

Adult average post-program weekly earnings N/A 

Adult average post-program weeks worked N/A 



Consi^f^nt with the intent of Congress, much of the responsibility for 
managing performance standards is placed at the state level, although the Depart- 
ment of Labor retains certain important duties. For programs funded under Title II 
of 3TPA, the basic federal role is defined in Section 106 as follows: 

(b)(1) ...for adult training programs under Title II... the Secretary shall 
prescribe standards on the basis of appropriate factors which may 
include (A) placement in unsubsidized employment, (BO retention in 
unsubsidized employment, (C) the increase in earnings, including 
hourly wages, and (D) reduction in the number of individuals and 
families receiving cash welfare payments and the amounts of such 
payments. 

(2) ...t^ ^cetary shall also designate factors for evaluating the per- 
' youth programs which... shall be (A) attainment of 

*^loyinent competencies, recognized by the private Industry council, 
(b> ilementary, secondary, and postsecondary school completion, or 
the equivalent thereof, and (C) enrollment in other training programs 
or apprenticeships, or enlistment in the Armed Forces. 

(*) The Secretary shall prescribe performance standards relating gross 
program expenditures to various performance measures. 

The standards must also include provisions governing the base period prior to partici- 
pation, a representative period after termination, and cost-effective methods for 
data collection. 

The authority of the Federal government to prescribe utilization of the 
standards is, however, clearly circumscribed by the Act. For example: 

(e) Each Governor may prescribe, within parameters established by the 
Secret try, variations in the standards.. .based upon specific economic, 
geograi^n'c, and denographic factors in the State and in service deliv- 
ery ar6o> *hin the State, the characteristics of the population to be 
served, ai ' le type of services to be provided. (3TPA, Section 106(e)) 
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To further support its emphasis on performance management, the Act also 
places performance standards prominently m the state and local planning process, 
and provides rewards and sanctions that are directly linked to local SDA perfor- 
mance. In the first case. Section 10^ requires SDAs to include in their annual plans 
performance goals as well as plans for the pr ^paration of an annual report that must 
address the extent to which these goals were achieved. In addition. Section 1 05(b)(1) 
requires Governors to approve SDA plans unless corrective measures for not meeting 
performance expectations have not been taken or are not underway. 

In the second case. Section 106(h) authorizes the Governor to provide tech- 
nical assistance to low performing SDAs and to apply sanctions if poor performance 
persists for at least two consecutive years. Further, Section 202(a)(3) makes avail- 
able six percent of the state allocation for distribution to those SDAs exceeding their 
performance standards, for technical assistance, and for services to hard-to-employ 
individuals. 

To fulfill the Secretary of Labor^s responsibilities under JTPA, Title IIA, the 
Employment and Training Administration (ETA) implemented regulations and a 
suggested framework for Governors to use for establishing performance standards. 
One of the most noteworthy features of this framework is the optional adjustment 
methodology for setting local performance expectations. This methodology builds 
upon the CETA experience by using regression techniques to account for factors out 
of the control of local service providers when setting standards. Since its initial 
implementation, howeveir, there has been growing emphasis placed on encouraging 
states to go beyond the model in setting local performance standards by considering 
those factors which may affect performance but are not included in the standard 
adjustment. In many respects, the current trend in JTPA performance management 
is headed toward the general approach used in the later stages of CETA which 
emphasized more of a balance between statistical forecasting and negotiation over 
key factors that influence participant employability. 

Although JTPA's performance standards system is expected to fulfill a 
critical management role, it can also create "perverse incentives" to serve the most 
job-ready applicants in short-term, low cost programs. To counteract these potential 
effects, JTPA contains at least five mechanisms, including: 

1. A requirement that 90 percent of all enroUees be disadvantaged; 

2. Targeting provisions requiring specified levels of service to youth, high 
school drop-outs, and welfare recipients; 

3. Authorization for the Governor to adjust SDA standards to account for 
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variations in participant characteristics and other factors affecting 
performance; 

The establishment of performance standards at a minimally acceptable 
level; and 

5. The availability of financial incentives that Governors can use to 
influence SDA behavior. 

Despite these efforts, it is possible that performance standards may have 
the effect of reducing services to hard-to-employ individuals or limiting the invest- 
ments that are made in participants. Indeed, this has been argued by practitioners, 
researchers, and representatives from the Congress. Many believe that the perfor- 
mance standard" system has shifted the attention of programs away from long-term 
employability development to the creation and management of a low cost, high 
volume service delivery system. 

Whether performance standards cause "creaming" and/or the delivery of 
low-cost services is one of the most controversial and confusing issues surrounding 
3TPA. While evaluative evidence on this issue is not yet available, at least four 
developments contribute to the concern.^ The first is widespread use of screening 
criteria for admission into JTPA programs. Program operators have been often 
found to screen out the most disadvantaged, through education and skill tests, in 
order to meet the performance standard provisions of their contracts. The second 
development is the relatively limited investment that JTPA makes in its partici- 
pants. For example, between July 198* and December 1985, the duration of service 
receipt was relatively short, even for the more expensive training services and the 
more difficult-to-serve participants. Overall, the median length of stay for Title IIA 
terminees was approximately 12.5 weeks; in job search assistance it was less than 
four weeks, while in classroom training, the median was roughly seventeen weeks. In 
addition, more disadvantaged enrollees did not receive substantially more intensive 



^Supoort for many of these concerns can be found in: Westat, Inc., Transi- 
tion Year Implementation of the Job Training Partnership Act , January 1985; Grinker 
Associates, Inc., An Independent Sector Assessment of the Job Training Partnership 
Act, Grinker Associates, Inc., New York, July 1986; Gary Orfield and Helene Sles- 
sarev. Job Training Under the New Federalism , Unemployment and Job Training 
Research Project, Chicago, Illinois, 1986; U.S. General Accounting Office, The Job 
Training Partnership Act: An Analysis of Support Cost Limits and Participant Char- 
acteristics , GAO, Washington, DC, November 6, 1985; and U.S. Department of Labor, 
Summary of JTLS Data for JTPA Title IIA and III Enrollment and Terminations 
During July-December 1985 , Washington, DC, May 1986. 
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services; youth were enrolled for about 12.5 weeks, while public assistance recipients 
received services for lUA weeks. 

The third development is the relatively low service share received by high 
school dropouts. In contrast to a 51 percent share of the eligible 3TPA population, 
service shares to this target group were 37 percent in PY8* and *0 percent in PY85. 
Although these shares closely conform to JTPA's targeting provisions, they can still 
be viewed as below the relative service needs of high school dropouts. 

The fourth and final development is growing concern voiced by the practi- 
tioner community that the performance standards system is simply constraining local 
discretion over whom to serve and what types of programs to offer. Four particular 
concerns about the performance standards system are advanced, including: 

1. The statistical adjustment models do not fully neutralize the risks 
associated with serving participant subgroups, and particularly AFDC 
recipients and other hard-to-employ groups. While the adjustment 
models do control for several important characteristics, they do not 
account for intra-group differences in characteristics that affect 
employability nor do they control for other key characteristics that 
also influence performance achievement, such as literacy and motiva- 
tion. Thus, program operators argue that they remain confronted with 
disincentives to serve hard-to-employ groups both within and across 
those already included in the adjustment procedure. 

2. Although the Department of Labor has developed a number of addi- 
tional procedures for adjusting performance beyond the model, anec- 
dotal evidence suggests that many states may not be utilizing them, 
and in those cases where they have been implemented, SDAs do not 
readily take advantage of them. While the reasons for this are unclear, 
many believe that they stem from the perceived complexity of the 
standards system generally, and the data and other requirements asso- 
ciated with these additional adjustment procedures in particular. 

3. The standards themselves and particularly the cost standards present a 
clear message that costs must be limited, thus constraining the type of 
investments that service providers can make and the types of individ- 
uals they can enrclK 

JTPA's overall emphasis on performance achievement has created a 
culture that places undue priority on the performance standards, at the 
expense of other program goals such as serving the hard-to-employ. 

These developments are not evidence of the effects of perfornr ,nce stan- 
dards per se; in fact, a study recently released by the National Commission for 
Employment Policy reports that with the exception of adult high scho. dropouts, 
other target groups including AFDC recipients, have been served in a manner pro- 
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portional to their representation in the eligible population. However, when com- 
bined with JTPA*s overriding emphasis on performance standards, tnese develop- 
ments have raised "red flags" and lead many to consider whether the program^s 
performance management system needs to provide greater stimulus for serving the 
hard-to-employ. 

At the same time, performance standards cannot be viewed in a vacuum. 
Other aspects of JTPA may also be contributing to concerns over creaming and 
limited program investments. Perhaps most important is that JTPA is not targeted 
on the hard-to-employ. Eligibility criteria for Title IIA programs are income-tested, 
focused almost exclusively on the economically disadvantaged, and sufficiently broad 
to permit local program operators to provide services to those who, in their judg- 
ment, can benefit from and are in most need of services. Thus, JTPA has a rela- 
tively large eligibility net to cast, from which it must select economically disadvan- 
taged individuals who are most in need of services and who can also benefit from 
service receipt. 

Related to this issue is the absence of unanimity regarding the characteris- 
tics of the hard-to-employ population. To date, neither the Department of Labor nor 
the Congress have explicitly defined the hard-to-employ population, making it diffi- 
cult to both judge existing service patterns and provide direction over whom to serve 
in the future. And even if the hard-to-employ were defined, it would remain to 
select those who could benefit most from program services. Absent a national 
definition, the very structure of JTPA would encourage variation on both a state and 
local level. 

In addition to targeting flexibility, JTPA^s limitation on the use of stipends 
and supportive services places greater reliance on individual self-selection among 
program eligibles. Participant motivation thus takes on a prominent role in JTPA 
and may well run counter to the desire of providing services to those with relatively 
limited incentives to seek work or assistance finding it. 

Like performance standards, the relative importance of these factors 
remains untested. They do, however, suggest that performance standards alone may 
not be at the cutting edge of the creaming and low-cost service problem. Modifying 
the performance standards system may thus be a necessary but not a sufficient 
condition for improving the level and quality of services to the hard-to-employ. 



See: Steven Sandell and Kalman Rupp, Who Is Served in JTPA Programs; 
Patterns of Participation and InterRroup Equity , National Commission for Employ- 
ment Policy, Washington, DC, February 1988. 
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Nonetheless, the seriousness of the issue has caused the Congress and the 
Department of Labor to reassess the performance standards system critically and to 
consider modifications to it. At the same time, the Congress and the Administration 
are also considering amendments to JTPA that would encourage services to AFDC 
recipients and other hard-to-employ groups, as well as legislation that would restruc- 
ture the welfare system and the manner in which training services are provided to 
public assistance recipients. Since several of the legislative proposals suggest the 
creation of performance standards to guide the provision of services to welfare 
recipients, an assessment of such standards can provide timely input to current 
policy deliberations. 

2.2 Performance Standards Management 

Modifying the performance standards system to further support the goal of 
serving AFDC recipients raises a number of substantive issues. While these issues 
are not new in a conceptual or research sense, their application to the hard-to- 
employ presents new challenges. 

The first challenge is developing a set of performance measures that are 
valid predictors—in both a statistical and intuitive sense— of JTPA*s objectives. 
Although substantial research has already been conducted on the validity of perfor- 
mance measures for employment and training programs, there are reasons for ques- 
tioning their application to training programs serving welfare recipients. First, each 
of the prior studies used as validation benchmarks impact estimates derived from 
quasi-experimental methods.^ Since it has been demonstrated that such estimation 



Each of the following validation studies relied upon quasi-experimental 
methods to derive benchmarks: 

Michael E. Borus, "Indicators of CETA Performance," Industrial and Labor Relations 
Review, Vol. 32, No. 1, October 1978. 

Katherine Dickinson, Terry Johnson and Richard West, An Analysis of the Impact of 
CETA Programs on Components of Earnings and Other Outcomes , SRI International, 
Menlo Park, CA, June 198^. 

Robert Gay and Michael Ec Borus, "Validating Performance Indicators for Employ- 
ment and Training Programs," The Journal of Human Resources , Vol. 15, No. 1, 
Winter 1980. 

V. J. Geraci and C. T. King, Employment and Training Program Performance; Long- 
Term Effects and Short-Term Indicators , Center for the Study of Human Resources, 
University of Texas at Austin, Austin, TX, September 1981. 
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techniques yield biased results of an often unknown magnitude and direction, it would 
be difficult to defend the use of such results in a new welfare reform program or in 
an effort to modify the existing JTPA performance measures. To the extent fea- 
sible, it would be desirable to reassess the existing measures using experimental 
data. 

Even if existing validation findings were substantiated using experimental 
impact estimates, it would also be necessary to determine if the indicators are also 
appropriate for programs targeted on welfare recipients. Since prior research has 
been based on the economically disadvantaged population, as opposed to welfare 
recipients, and because these studies have used earnings, not welfare dependency, as 
the benchmark measure, there is no a priori reason to assume that their findings can 
be fully generalized to a targeted segment of the disadvantaged population. In fact, 
recent evidence would suggest a potential problem and the need for further investi- 
gation. 

To see this, consider the information presented in the following three 
exhibits. Exhibits 2.3 and 2A display the most recent validation findings for 
employment and training programs, using earnings as the benchmark. As can be 
seen, nearly every measure tested is significant and has the expected positive sign, 
suggesting that JTPA^s current performance indicators are valid management tools. 
Given their predictive validity, one could assume that they are appropriate for 
targeted welfare programs. However when we consider the findings in the next 
exhibit, such an assumption becomes tenuous. 

Exhibit 2.5 presents recent validation findings obtained by the Manpower 
Demonstration Research Corporation (MDRC) based on data collected from their 
work-welfare demonstrations. Using experimental earnings and welfare savings 
impacts, the researchers attempted to validate whether employment or welfare 
status soon after program termination were good predictors. The results are disap- 
pointing and suggest that at least two intuitively appealing performance measures 
may be inappropriate for targeted welfare programs. As the study summarized: 

This conclusion— which runs counter to common wisdom— simply reflects the 
fact that the magnitude of the program effect on finding a job or leaving 
welfare is greater for some groups of individuals than others. This does not 
imply that programs should stop trying to help all people in the caseload 
find jobs and leave welfare. It does mean that judging programs on the 



Jeffrey Zornitsky, et al.. Establishing a Post-Program Performance Management 
System for Employmer'^ and Training Programs , Abt Associates Inc., Cambridge, MA, 
1985. 
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Exhibit 2.3 

Title IIA Validation Correlations for Adjusted Level Measures Using CLMS Data 
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SOURCE: Zornitsky, Jeffrey, et al. Establishing a Post-Program Performance Management System for Employment and Training Programs , Abt 
Associates Inc., Cambridge, MA, December 1986. 
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Exhibit 2A 

Validation Correlations for Adjusted Title IIA Change and Level Performance Measures Using CLMS Data 
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SOURCE: Zornitsky, Jeffrey, et al, Establishing a Post-Program Performance Management System for Employment and Training Programs , Abt 
Associates Inc., Cambridge, MA, December 1986. 



Exhibit 2.5 

Recent Findings on the Validity of Candidate Performance Measures for 



AFDC Recipients Receiving Employnnent and Training Services 



Program and Welfare Status 



San Diego, AppI icants 
Ba 1 1 i more , App I i cants 
Baltimore^ Recipients 
Baltimore, Al I AFDC 



Val idity of Job Entry 



Earnings Gain 



poor 
poor 
poor 
weak 



Wei fare Savings 



yalldlty of Welfare Status 



Earnings Gain 



weak 
poor (fair) 

weak 
weak (fair) 



fair 
poor 
fair 
fair 



Welfare Savings 



weak 
poor 
good 
fair 



NOTES: This table summarizes the correlations between the designated indicator and the earnings gains or welfare savings. The 
following symbols are used: 

* Good, indicates a correlation that has the correct sign and is statistically significant. 

• Fair, indicates a correlation that has the correct sign but is not statistically significant. 

• Weak, indicates a correlation that has the wrcr.g sign but is not statistically significant. 

♦ Poor, indicates a correlation that has the wrong sign and is statistically significant. 

SOURCE* Oaniei Friedlander and David long, A Study of Performance Measures and Sub-Group Impacts in Three Welfare Employment Programs , 
Manpower Demonstration Research Corporation, New York, N.Y., Merch 1987. 
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basis of these outcome measures— without considering differences in case- 
load characteristics and economic conditions— is unwise. It is quite pos- 
sible, for example, for a program with a relatively low placement rate in a 
poor labor market to have greater impacts than another program with a 
more job-ready caseload and more placements. The analysis also shows that 
this conclusion does not change when longer-term employment rates are 
substituted for immediate job entries. 

While these findings were obtained from a particular demonstration pro- 
gram, they do cast some doubt on the use of gross, short-term performance mea- 
sures. At a minimum, the results suggest that more validation research, using exper- 
imental data from training programs targeted on welfare recipients is quite desir- 
able. The findings also imply that welfare-related performance standards should be 
adjusted for the background characteristics of individuals. While it can be argued 
that the existing adjustment model accomplishes this, the current adjustments do not 
account for at least one key characteristic which improved predictive validity in 
MDRC's research-that being prior work history. Recall that MDRC found that the 
predictive validity of the two candidate measures improved substantially when they 
were adjusted for earnings during the year prior to program participation. Thus, 
even the existing welfare adjustment model may be inadequate as currently specified 
to reduce creaming incentives within the AFDC population substantially. 

It would thus appear that indeed, more research is needed to identify valid 
performance measures for targeted welfare programs. While research to date pro- 
vides many consistent findings, it remains to tost their robustness with experimen- 
tally derived impacts of welfare reductions for public assistance recipients served in 
employment and training programs. 

The second major challenge is to design a methodology and management 
strategy that can be used to set performance standards. Such a methodology needs 
to not only address statistical forecasting criteria, but it must also be sufficiently 
informative, practical, and flexible to maximize the targeting of services to recipi- 
ents, and especially to hard-to-employ individuals. As discussed above, the MDRC 
findings suggest that the existing model for adjusting the adult welfare entered 
employment rate may be inconnplete with respect to background characteristics. If 
this is the case, then the adjusted standards may be biased upward, at least in terms 
of serving long-term, hard-to-employ public assistance recipients. Unless additional 



Daniel Friedlander and David Long, A Study of Performance Measures and 
Sub-Group Impacts in Three Welfare Employ m'ent Programs , Manpower Demonstra- 
tion Research Corporation, New York, N.Y., March 1987, pp. xii. 
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data collection requirement*^ are foi thconning, new ways for adjusting standards may 
^ell be required. 

Establishing a flexible and informative performance standards setting 
process needs to go beyond a complete, valid adjustment model. Funding agents and 
program operators need assistance and information to determine how to set their 
standards and take a::count of difficult-to-measure obstacles faced hy prospective 
participants. The best of models cannot be expected to include all important explan- 
atory variables, due to both measurement and data collection limitations. Thus, the 
standards setting process needs to be viewed in a collaborative context, where fund- 
ing agents, such as the Department of Labor and states, provide technical assistance 
and information in order to inform the local decision-making process. 
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SERVING WELFARE RECIPIENTS AND THE HARD-TOEMPLOY 



In the 1980s, welfare reform has again become an issue of national prom- 
inence. Ongoing debates at the federal and state levels have resulted in a broad 
cons-nsus across the political spectrum that work opportunities must be an integral 
part of the welfare system, and that the system of public welfare in the United 
States must be structured to promote employment rat^ t than dependency. In the 
nine states that we surveyed, it is evident that there is substantial activity aimed at 
pursuing this objective. The majority of these states are engaged in implementing 
ambitious efforts to target services to those most in need (individuals with multiple 
barriers to employment) and to deliver a bread array of basic education, vocational 
skill training, job placement, and support services. 

The impetus for these initiatives stems from the Reagan Administration's 
proposal in 1981 to eliminate the WIN program and require states to establish manda- 
tory workfare programs called Community Work E/ ;erience Programs (CWEP). The 
concept of workfare was first used in state and local general relief programs as early 
as the 1930s, but was generally prohibited for the AFDC program. Responding to the 
Administration initiative in the Omnibus Budget Reconciliation Act of 1981 (OBRA), 
the Tax Equity and Fiscal Responsibility Act of 1982, ^'nd the Deficit Reduction Act 
of 198^ (DEFRA), Congress authorized the states the option of states' operating a 
variety of work-related programs and to make participation mandatory for AFDC 
recipients. Four major options are available: 

WIN Demonstration Programs. This permits states to establish alterna- 
tive programs to WIN, administered by the state welfare agency rather 
than in conjunction with the SESA, either on a state-wide or substate 
basis. States enjoy broad discretion in designing programs and allocat- 
ing resources. Services may include vocational counseling, basic skills 
remediation, job development, and job search assistance. States also 
have the option of consolidating services for AFDC and General Assis- 
tance recipients by changing the WIN structure to make it compatible 
with state GA work programs. The purpose of this option is to give 
states maximum flexibility in developing a work/welfare policy. 

• Community Work Experience Programs. This permits state agencies to 
establish workfare programs in which recipients are required to work 
without pay each month at public or non-profit agencies. The hours of 
work to be required are calculated cy dividing the amount of the family 
public assistance grant by the minimum wage. The purpose of this 
option is to require welfare recipients to "earn" the value of their 
grants, thus creating a disincentive to long-term dependency while also 
offering an opportunity to acquire work experience. CWEP partici- 
pants must perform work with a useful purpose and may not be used to 
substitute for regular workers. 
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Work Supplementation Programs. This permits the state agency to use 
a recipient's welfare grant to subsidize a job. The purpof^ of this 
option is to encourage employers to hire AFDC recipients by reimburs- 
ing them for a part of the wages paid. In this sense, work supplementa- 
tion operates much like on-the-job training. Jobs may be developed 
with public, private non-profit, or private employers. The objective of 
the program is that employers hire the subsidized worker at the end of 
a four- to six-month period of subsidy. 

• Job Search Progiams. In this option, job search may be required of 
applicants as well as of recipients as a condition of eligibility. Up to 
16 weeks of job search may be required in the first year of assistance, 
and up to 8 weeks in each subsequent year. Required activities may 
also include orientation to work, group job search classes (job clubs), 
and referrals to specific jobs. 

The states which we interviewed have embraced various combinations of 
these options to shape their demonstration and reform efforts. In nearly all 
instances, these initiatives involved new, additional state funds and the combined 
resources and efforts of both JTPA and departments of public welfare. Moreover, in 
some states, JTPA has taken independent steps to strengthen its responsiveness to 
the welfare population. Targeted use of six percent incentive funds, for example, as 
well as local program coordination and flexible procedures to set numeric perfor- 
mance goals, are key tools that some state JTPA offices have relied upon to stimu- 
late service delivery to the welfare population. 

The purpose of this chapter is to create the program and policy context for 
understanding how performance standards may affect the delivery of services to the 
welfare population. Since performance standards are managed within a complex and 
evolving set of programs, it is important to identify the key factors that influence 
whom they serve and how they are designed. We begin the chapter with a discussion 
of key policy and program issues that surfaced during our interviews and which have 
a bearing on the development of effective program services for the AFDC popula- 
tion. We then present a review of the salient features of new and emerging state 
initiatives to serve AFDC recipients. 

3.1 Key Program and Policy Issues in Serving AFDC Recipients 

The majority of states that we interviewed indicated that targeting welfare 
recipients and identifying the most appropriate types of services to provide to them 
were policy issues they have had to address. There was, however, noticeable varia- 
tion in the degree of sophistication and the stage of development in which states 
have actually addressed these issues. Within this context, we found that states 
generally fell into one of two major categories. The first group includes those that 
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are well ensconced in full-fledged efforts to improve the level and quality of services 
for the welfare population. For the most part, these states have implemented 
efforts that center on reform initiatives and include specific targeting criteria, 
provisions requiring coordination between JTPA and departments of public welfare, 
and enriched program designs, emphasizing remediation services as well as more 
output-oriented services such as occupational skills training. 

The second group includes states that are just in the initial stages of broad 
efforts to improve service delivery and develop targeted approaches to serve the 
AFDC population. In these states as well, however, most of the attention and devel- 
opment activity is aimed at demonstration and reform initiatives, rather than at the 
ongoing system of existing services. 

A major factor motivating these activities is top-level gubernatorial sup- 
port. In nearly every state we interviewed, respondents indicated that welfare 
dependency was a major political issue for governors. While the form that this 
leadership took ranged from new state legislation, to special demonstration pro- 
grams, to inter-agency task forces composed of key state officials, it became clear 
during the interviews that growing public concern with the issue of welfare depen- 
dency had motivated several states to develop their own brand of welfare reform. 

While strong public support for reforming the welfare system has indeed 
supported new, innovative state-based efforts, it has al?o introduced a tension 
regarding the expected outcome of these initiatives. During our interviews we noted 
a conflict that many state officials were increasingly faced with as they attempted 
to improve the level and quality of service delivery to AF^^DC recipients. On the one 
hand, respondents indicated considerable public concern over whether funds were 
being spent well. In many cases, this was translated into political pressure to demon- 
strate on both the state and local service provider level "more bang for the buck." 
Success in this context was usually interpreted quite narrowly to mean quick and 
easily quantified results, such as large numbers of low-cost placements. One respon- 
dent from a state welfare office told us that his office emphasizes working with the 
relatively "easy-to-serve" first because county welfare offices want to "look good on 
paper" and need to demonstrate quick success. 

On the other hand, many of the same state respondents recognized that 
serving the AFDC population effectively requires significant, long-term investments 
of public resources that may not demonstrate visible results in the short run. This 
perspective was viewed as counter to what many officials believed was necessary to 
garner public support, and in their view, created an untenable conflict. It nonethe- 
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less created clear incentives foi officials and program operators to engage in short- 
term program services that emphasized low costs and immediate placement. For 
this reason, many states viewed consensus building and clarification of the role of 
work and welfare programs as key to their reform efforts. 

Regardless of the status of state reform efforts, we observed a general 
consensus that the current structure and mix of employment and training programs, 
by themselves, cannot be expected to resolve the multiple barriers to employment 
that many AFDC recipients, and especially those who are hard-to-employ, often 
face. One respondent from a state welfare department told us, for example, that: 

Simply modifying 3TPA is not going to solve the whole problem of effec- 
tively providing employment and training services to hard-to-employ AFDC 
recipients. It will help, but it will not be a panacea because the absolute 
level of 3TPA resources is simply not sufficient to serve hard-to-employ 
clients. 

Although most respondents from welfare offices indicated that increased 
funding was critical to improving the quality of services to AFDC recipients, most 
3TPA respondents did not share this perspective. All respondents, however, cited the 
lack of interagency coordination at the state and especially local levfels, as an imped- 
iment to successfully serving the welfare population. There was broad agreement 
that more effective use could be made of existing employment and training funds 
through increased coordination. Most respondents recommended ti.at states need to 
focus attention on developing joint JTPA-walfare solutions to enable v'elfare recipi- 
ents to become self-sufficient. The importance of cc-ordination can be seen, for 
example, in a recent report prepared for the Michig^in Job Training Coordinating 
Council. Commenting on coordination, the report says: 

JTPA was not established as a comprehensive job training system and it is 
apparent that the multiple needs of the hard-to-serve cannot be met exclu- 
sively by 3TPA. Furthermore, pushing each parallel system to be compre- 
hensive is not cost-effective and does not guarantee the needed level of 
service. It is only across all systems, such as education, vocational educa- 
tion, employment service, social services, job training, etc. that the mul- 
tiple needs of hard-to-serve individuals can be met. Coordinated outcomes 
and operations at state and local levels are necessary t*^ provide integrated 
service delivery capable of m.eeting these multiple needs. ^ 

Our respondents cited several factors contributing to the lack of coc.uirid- 
tion between 3TPA and Welfare. One centered upon existing differences, in target 
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group focus. While welfare departments are targeted exclusively on the AFDC 
population, the JTPA program is more broadly targeted on all disadvantaged indi- 
viduals, and must structure its services to meet the needs of a wide-ranging popula- 
tion. To the extent that JTPA increasingly targets on AFDC recipients, it runs the 
risk of losing key elements of its communAty support and constituency. Another 
factor frequently mentioned was the perspective that JTPA is designed cO provide 
employment and training, not education services. Since many AFDC recipients often 
require remediation services to successfully participate in other programs, the JTPA 
emphasis on training per se limits its ability and perceived responsibility to serve the 
welfare population. 

JTPA's performance standards system was also mentioned, especially by 
JTPA respondents, as inhibiting coordination. Many respondents viewed the stan- 
dards as placing substantial pressure on SDAs to produce a high volume of low-cost 
placements. Because serving the AFDC population often requires long-term, costly 
investments, many believed that SDAs faced disincentives to serve this group, on 
either an independent or a coordinated basis. 

Last, but significant, were institutional barriers stemming from the exi^ 
tence of separate administrative and service delivery systems and differences in 
opinion over the proper relationship between work and welfare. While these factors 
varied across the states we interviewed, they were frequently mentioned and 
described in the context of turf battles. While each side was readily able to identify 
what the other could do better, both tended to be unwilling to give up much in the 
process. 

Even if coordination, limited political pressure, and enriched program 
services could be achieved, our results indicated that other issues would remain to be 
resolved. Almost unanimously, our respondents pointed to the need to augment and 
increase flexibility surrounding the availability and use of supportive services. In 
nearly every interview, this was mentioned as a key factor inhibiting service delivery 
to AFDC recipients and especially the hard-to-employ. Similarly, several respon- 
dents rep ealed sc^ ^ measure of difficulty defining the !iard-to-employ and identify- 
ing ho c best provide new, enriched and comprehensive services for them. For 
these .asons, we observed only a limited effort in some states to implement full- 
scale S'.zXe-mde demonstration and reform efforts. In these states, the demonstra- 
tions were intended to provide important lessons and guidance on how to best tackle 
the work -welfare issue. 
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It would thus appear that creating effective program responses for the 
welfare population involves several complex program end institutionai factors. 
Clarity over targeting expectations, flexible and comprehensive program services 
through inter-agency coordination, and the availability of adequate levels of suppor- 
tive services are, for example, among some of the more important ingredients of 
effectively operating programs. Improving the responsiveness of the employment 
and training system will thus have to consider these factors as well as modifications 
to the performance standards system. 

3.2 EmergiMR Policies and Programs to Serve AFDC Recipients 

Throughout all the interviews we conducted, we observed substantial activ- 
ity aimed at eloping new policies and programs designed to strengthen service 
delivery to AFDC recipients and to target such services on those identified as hard- 
to-employ. We also observed noticeable interstate variations in the level, types, and 
degree of sophistication of these efforts that can be explained by varying degrees of 
political pressure, prior experience with demonstration initiatives, the degree of 
flexibility in the existing service delivery system, and the ability to clearly identify 
the characteristics of hard-to-employ individuals. 

Several states are, for example, relying upon their performance standards 
system to encourage service delivery to AFDC recipients and especially those identi- 
fied as hard-to-employ. In particular, they are relying on theii" six percent policies 
and further adjustments to the Department of Labor's models to further remove any 
disincentives local SDAs might face in targeting and serving these groups. In other 
cases, we observed the use of eight percent JTPA education funds to augment the 
availability of remediation services. Finally, in several states (including Pennsyl- 
vania, Florida, and Michigan) new statutory requirements have been enacted to 
provide improved employment and training services to welfare recipients. But 
perhaps the most significant vehicle used by states to expand and improve the deliv- 
ery of services to AFDC recipients are state-wide welfare reform and other pilot 
demonstration programs. 

Recent surveys of state work-welfare programs by the General Accounting 
Office and the Congressional Budget Office indicate a substantial level and diversity 
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of state-based welfare reform initiatives. The GAO study found, for example, that 
in 1985, 38 states had selected one or more new program options, including 25 WIN 
demonstrations, 20 Community Work Experience Programs (CWEPs), 6 job search 
programs, and 10 work supplementation programs. A total of 2^ of these states 
chose to operate more than one option. GAO further found that states were inter- 
ested in testing new approaches, sometimes operating one or more pilot projects, and 
sometimes starting with one program and modifying it over time. The general trend 
observed was towards larger, more comprehensive programs with a rich mix of 
services, including both education and training. 

Each of the nine states that we surveyed has recently begun to introduce 
one or more new forms of welfare-to-work programs. These efforts are occurring on 
a state -wide basis in Pennsylvania, Florida, Massachusetts, Illinois, Michigan, New 
York, and New Jersey through a mixture of broad welfare reform initiatives as well 
as smaller pilot demonstration projects. By contrast, Texas and Ohio are engaged in 
smaller-scale efforts. Regardless of the scope of each effort, however, their key 
goals ore fundamentally similar, although not identicak First, there is a recognition 
that hard-to-serve clients need an "enriched" mixture of program services which 
address both their lack of basic skills and their need for vocationally-specific skills. 
Second, there is an expectation that outcomes for this group will improve with a 
more intensive concentration of services. Third, there is a strong emphasis on inter- 
agency coordination at both the state and local levels. 

At the state level, the demonstrations typically involve the state depart- 
ment of welfare, JTPA, social services, and in some cases, the state department of 
education. These agencies have worked together to identify respective roles and 
responsibilities and efficient means of making their programmatic resources avail- 
able, and to define procedures for directly monitoring program activities. In several 
cases, these groups have come together as part of top-level interagency task forces 
created to address issues related to welfare dependency. In other cases, they have 
been brought together as a result of the demonstration projects and remain involved 
through that vehicle. 

For example, in Project Chance, Illinois* major state v/elfare initiative, 
state-level coordination is a keystone. The initiative's designers recognized from the 



U.S. General Accounting O.fice, Work and Welfare: Current AFDC Work 
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beginning that a joint effort is essential to achieve the Project Chance goal of plac- 
ing over 100,000 welfare recipients in jobs by June 1988; success could not be real- 
ized single-handedly by the Department of Public Aid. Therefore, in December 1985 
Project Chance was inaugurated by the Governor together with the Directors of the 
Departments of Public Aid, Employment Security and Commerce, and Community 
Affairs. Specific details of the mandated interagency cooperation include the fol- 
lowing: 

Project Chance takes full advantage of the Department of Employment 
Security's (DES) computerized job service in referring job-ready welfare 
recipients. Recipients previously served directly by DES are now part of 
the Project Chance caseload, and a new interagency contract has been 
executed for PY 1987 which established performance objectives for DES 
that will result in increased referrals and job placements. Recipients in 
need of job training are referred to Job Training Partnership Act (JTPA) 
programs, which are mandated to help reduce welfare dependency. Recipi- 
ents needing more basic sducation are provided with literacy training 
through the Illinois Literacy Council and adult basic education, and GED 
programs offered by the State Board of Education and Public Aid. 

Similarly, Florida's statewide welfare reform initiative. Project Indepen- 
dence, is also designed to enable the state to design and implement innovative job 
training strategies for hard-to-serve welfare recipients. The Department of Health 
and Rehabilitation Services (DHRS) has been designated as the single state agency 
responsible for the planning, integration, and coordination of employment-related 
services for welfare recipients; seven other sta^e agencies as well as PiCs and local 
education agencies are mandated to cooperate with the DHRS in carrying out this 
mission. A state-lovel interagency task force created a management plan which 
forms the core of Project Independence; that is, to create interagency pilot projects 
which will provide job-specific training for welfare recipients, often using grant 
diversion to support their placement in guaranteed and targeted jobs. The goals of 
Project Independence are: 

• To increase the number of public assistance recipients entering 
employment, 

• To provide screening services to participants in order to determine job- 
readiness, and 

• To increase training and educational opportunities through JTPA and 
local education agencies for welfare recipients who have educational 
and work skills deficiencies. 



^ Project Chance Annual Status Report , Springfield, IL, 1986. 
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At the local level, the program design of the demonstrations also encour- 
Jiges cooperation between public agencies such as county welfare offices, SDAs, local 
Employment Service offices, and local education agencies. The first rationale for 
such coordination is to facilitate client flow between SDAs, county welfare offices, 
and local education agencies. The idea here is to facilitate welfare recipients* 
receipt of the basic services that they need— employment and training, public assis- 
tance, and educational services— and to prevent them from /becoming discouraged in 
their efforts to become economically self-sufficient. This will allow for more effec- 
tive allocation of already-available training and educational program slots to those 
welfare recipients and other hard-to-serve individuals who could most benefit from 
such opportunities. Thus, the intent of institutionalizing such cooperation is to avoid 
the necessity of creating a wholly new employment and training system for welfare 
recipients. 

For example, Michigan has begun an interagency program initiative tar- 
geted towards the hard-to-serve of whom welfare recipients are a major subpopula- 
tion. To implement the hard-to-serve policy statement adopted by the Michigan Job 
Training Coordinating Council, state agencies have combined resources and have 
formed a partnership with local agencies in order to achieve several goals specified 
in a 1987 Hard-to-Serve Program Initie'.ive. The goals of this initiative are: 

• To increase the employment, training, and related service opportunities 
for the hard-to-serve through existing programs. 

• To increase the number of hard-to-serve individuals who become 
employed and remain employed as a result of acquired skills and com- 
petencies. 

To develop an integrated system of service delivery which is transpar- 
ent and therefore easily accessible to individual users and which makes 
maximum use of existing resources to identify and meet their needs. 

To support this initiative, Michigan has committed $1.2 million to support 
eight small-scale demonstration projects. One such project: 

v/ill utilize a combination of state and local resources to establish a multi- 
purpose training facility which will provide comprehensive training and 
support services to hard-to-serve individuals residing in three local coun- 
ties. The center, which will house literacy, adult basic education, general 
educational training. Head Start, and an array of personal development 
services, represents a coalition effort on behalf of the educational and 
human service providers in northeastern Michigan. The major goal of the 
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program is to provide primarily female, single parent clientele (many of 
whom are AFDC recipients) with the intermediate steps necessary to access 
traditional employment and training programs. 

The second rationale is to avoid duplication of effort in such activities as 
eligibility determination. In Texas, for example, the state has arranged that in order 
to determine the JTPA eligibility of a welfare recipient, SDAs will first obtain 
required data on AFDC recipients from local welfare offices and then only if neces- 
sary ask for more information. The objective here is to minimize the paperwork 
burden placed on the recipient who, previously, would have to fill out welfare-related 
paperwork^ and then repeat much of the same at the SDA intake office. 

In summary, in every state which has implemented major efforts to more 
effectively address the training and educational n^teds of welfare recipients, inter- 
agency coordination is seen as a key ingredient oi the vitality and success of these 
efforts. Top-level policy directives and technical assistance c,: both necessary to 
create the context for such efforts. The commitment of line managers in the rele- 
vant agencies is acknowledged to be crucial for achieving successful coordination 
outcomes. 

Just as these demonstration programs emphasize a high degree of inter- 
agency coordination, they are also targeted cn selective segments of the AFDC 
population believed to reflect the hard-to-employ population. Exhibit 3.1 presents 
the specific targeting provisions contained in the welfare reform initiativ s imple- 
mented in the states we surveyed. Several notable features emerge. 

First, ii.ere is noticeable diversity among the states with respect to the 
characteristics of the hard-to-employ. For example. New Jersey is primaril> focus- 
ing on welfare applicants and recipients with children aged 2 or older. In contrast, 
Illinois has idei.afied a broader array of target groups, " lading female AFDC 
recipients with children over age 6, male General Assistance recipients, AFDC 
recipients who live in public housing, and young parents under age 21. 

Second, the emphasis of these targeting policies varies between groups that 
are most at risk of becoming long-term welfare dependent (e.g., teen mothers) and 
groups that are thought to have low employabilit.y potential (e.g., high school drop- 
outs, the homeless, individuals with severe literacy problems). Although states -^o 
recognize that all groups which are the focus of targeting efforts typically s! 
common barriers to employment, such as lack of specific vocational skills or prior 
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Exhibit 3.1 



Characteristics of Hard-to-Serve Target Groups 
in State-Based Welfare Reform Initiatives 



state 

Pennsylvania 



New York 



Texas 



Program Name (Type) 

Single Point of 
Contact (SPOC) Demon- 
stration 



Transitional ly Needy 
(TN) Demonstration 

Comprehensive Employ- 
ment Opportunity 
Support Centers 
(CEOSC) Program 
(state-wide) 



Unnamed program 



Kard-to-Serve Target Groups 

Single parents with children under age 6 

Single parents who have been on public assis- 
tance for at least two years 

Individuals with less than a 6th grade educa- 
tion 

Individuals with severe literacy problems 

Employable recipients of General Assistance 
who have multiple barriers to employment 

Public assistance rec;p»ents vHth children 
under age 6 

Current or former teen parent who is on public 
assistance 

An individual who has been on the welfare 
rolls for two years or longer (Note: New York 
recently changed its definition of long-term 
receipt from four years to two years, in 
response to recent research on the dynamics of 
wel fare receipt, ) 

16-17 year old dropouts who are mandatory 
(WIN) registrants 

Newly certified welfare recipients 

Individuals who have received AFDC for 2+ 
years 

Individuals who have received AFDC for 2 years 
or less 
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Exhibit 3.1 (continued) 
Characteristics of Hard-to-Serve Target Groups 



in btate-Based Weltare Ketorm initiatives 



state 



Program Name (Type) 



Hard-to-Serve Target Groups 



Michigan 



Michigan Opportunity 
and Ski 1 1 Traini ng 
(MOST) Program (state- 
wide) 



Prih.dry wage earner in an AFDC-UP household 
(although the spouse can be a voluntary MOST 
registrant) 



Teens who are age 16 or older, not in school 
full-time, and part of an AFDC case 

An individual who is economically disadvan- 
taged 

An individual who has been unemployed 2+ years 
or never been employed 

An individual who Is in need of functional 
literacy or employabi 1 1 ty characteristics or 
supportive services 

An Individual whose employabi I ity development 
plan states that the length of time or cost of 
training leading to employment exceeds that 
received by the average person enrolled in a 
designated area. 



Parents aged 18 to 21 with children under age 
6 

Handicapped indi vidual s 



Ohio 



Supported Work Demon- 
stration 



Individuals who have been on public assistance 
for 3 to 6 years 



Massachusetts 



ET Choices (state- 
wide) 



Individuals receiving AFOC for 2+ years 



Teen parents 



Hispanics 



Public housing residents 




Exhibit 3,1 (continued) 



Characteristics of Hard-to-Serve Target Grjups 
m btate-Hased Weitare Ketorm initiatives 



state 



Program Name (Type) Hard-to-Serve Target Groups 



Florida 



1 1 1 inol s 



Project Independence 
(state-wide) 

Project Chance (state- 
wide) 



Female AFDC recipients with children age 3 and 
ol der 

Female AFDC recipients with children over age 
6 



New Jersey 



Real izing Economic 
Achievement (REACH) 
Program (state-wida) 



Male General Assistance recipients in Chicago 
AFDC recipients in public housing 
Parents under age 21 

Welfare applicants with children age 2 or 
ol der 

Welfare recipients with children age 2 or 
ol der 
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'work history, the differences between their targeting policies reflect the special 
barriers to employment faced by a particular group. So, for example, the objective 
of targeting subgroups who have a high potential for long-term welfare dependency is 
to assist with their extensive family responsibilities as well as their educational and 
skill development. The JTPA and welfare officials whom we interviewed are well 
aware that such individuals will need a rich mix of program components emphasizing 
child care and other supportive services. By contrast, high school dropouts and other 
subgroups with low employability potential face particular barriers to employment 
based on their lack of basic reading and computation skills. For such individuals, an 
effective program mix will emphasize remedial education services, for example, 
instead of or in addition to child care services. 

Overall, while we observed diversity among the targeting policies, it is 
important to point oui that the majority of states' policies are targeted toward 
female welfare recipients who are exempt from mandatory WIN particir -ion 
hicause xheir children are under age six. However, the age of the youngest child 
which acts -s a trigger for mandatory participai . i varies. For example, in order to 
target teen mothers (to prevent dropouts and second pregnancies), Florida, Uilnois, 
and New Jersey have each requested a federal waiver to lower the age of. youngest 
child for mandatory participation from age six to age three. The stated goal of this 
and similar targeting policies is to reduce the likelihooa of long-term welfare depen- 
dency. 

Service delivery approaches in these initiatives have no particular unifor- 
.•nity across states or between demonstration projects within a single state, based on 
our interview results. Flexibility, innovation, case management services, and an 
enriched program mix are, however, typical hallmarks. 

Our respondents observed that flexibility in program design is crucial to 
allow for the adaptation of basic program design elements to ar'iress the program- 
matic challenges created by particular local conditions or clientele. Typically, the 
authorizing legislation and/or policy directive (e.g., via an RFP) sets the core fea- 
tures of a program but allows a broad level of discretion in settmg the range, mix, 
and level of program services. 

for example, in New Jersey the focus of the REACH initiative is on locally- 
based planning and operation; the state does require, however, that the counties 
provide certain types of services, e.g., case management services, job search assis- 
tance, child care services, training-related expenses, and community work experi- 
ence programs. Within the context of this generic program mix, however, counties 
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have the flexibility to choose what services to emphasize. For example, they can 
choose the duration of job search assistance activities and what percentage of clients 
must go through JSA. The initiative's underlying philosophy is to rely on an indi- 
vidualized case management approach to determine what services best meet an 
individual client's particular needs rather than simply implementing a "blanket ser- 
vice delivery approach." 

Pennsylvania has also adopted flexible service delivery as a key theme in its 
new welfare reform initiative. The Joint Jobs Initiative is the legislative center- 
piece of the state's efforts to influence the types of services provided to welfare 
recipients; the SPOC (Single Point of Contact) and TN (Transitionally Needy) Demon- 
strations are the programmatic expressions of the Initiative. The underlying philo- 
sophy of the Initiative is that "people need different services based on their particu- 
lar background." Therefore, the state is encouraging the demonstration sites to 
develop a wide range of program options for AFDC clients; for example, services 
provided through SPOC include occupational skills classroom training, ^educational 
services, paid work experience, on-the-job training, educational services, job devel- 
opment/placement, supportive services, and case management services. Although 
each category of training has been defined in the SPOC RFP and there is a $5,000 
per participant cost limit, the length of training has not been specified. 

A key feature of many of the demonstration projects that we noted is an 
emphasis on client/staff contact, which is both more extensive and more intensive 
than in traditional welfare department programs. Case management services are the 
primary vehicle for this enhanced client/staff contact. Case management services 
are designed to prepare comprehensive employment plans for participants, to assure 
access for the necessary training and support sei vices, and to provide support and 
counseling during program participation and also after job placement. 

Several of our respondents noted that this enhanced one-on-one contact v's 
designed to counter the inability of traditional welfare-related employment and 
training programs to provide more than minimal personalized otaff attention. One of 
our welfare department respondents observed: 

A typical JTPA program doesn't do case management, so case management 
will be a key component in nelping to achieve the performance measures in 
[new state welfare reform initiative] by, for example, helping to keep 
dropouts low. That is, bard-to-serve welfare clients have high dropout 
participation rates in traditional employment and training programs. So, 
case management as well as supportive services will help these parti^.ipants 
stay in the program and achieve success. 
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As an illustration, one welfare department official provided the rationale 
for her state's new focus on case management services. Prior to its welfare reform 
initiative, the 160,000 clients in the state welfare department's employment and 
training programs were processed through 327 welfare department staff twice annu- 
ally. Further, these same clients were getting only about 20 minutes of service per 
year. To address such superficial service delivery, the new demonstration projects 
emphasize the case management approach as a vehicle to facilitate more effective 
client/staff interactions. 

State-wide case management services are now being provided in Massachu- 
setts' ET/Choices Program and Florida's Project Independence. Pennsylvania and 
New Jersey have also incorporated case management services into their major wel- 
fare reform initiatives. As described in Florida's Project Independence: 

The purpose of case management is to enr.ble the case manager and the 
participant together to recognize the participant's capacity for self- 
reliance, and to develop an action plan which leads to self-sufficiency. 
Case management includes four major functions: assessment, development 
of an Employment Plan Contract, arranging and coordinating activities and 
resources necessary to carry out the plan, and followup to make sure the 
services and activities were provided. Assessment is to be used to increase 
the participant's motivation to participate in program activities by focusing 
on the participant's strength and past experiences and by offering available 
program options in a positive manner. 

Our respondents observed that the d -nonstrations typically include 
enriched program components. These are designed to more comprehensively address 
the multiple barriers to employment that many welfare recipients possess. Typi- 
cally, there is a broad and varied mix of employment and training services including 
assessment of needs and abilities, remedial education for those reading LjIow a 
sixth-grade level, work experience, classroom vocational skills training, on-the-job 
training, job development, job search assistance, and job placemt ' assistance. 

For example, our Michigan respondents told us that the viight demonstration 
projects targeted to hard-to-serve clients, particularly welfare recipients, are 
de gned to provide a sequence of program components which are longer-term and 
mc ^ intensive than traditional employment and training services available to vvel- 
fare recipients. The expectation of the state, in funding these projects, is that these 
clients will participate in the demonstration projects for a longer period of time as 
compared to the traditional prog^amSf 

Across all the major demonstration projects, we noted that support services 
such as child care and transportation are required to be provided in sufficient quan- 
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tity as core services during program participation. These are viewed as crucial to 
the ability of welfare recipients, particularly hard-to-serve subgroups of the AFDC 
population such as young mothers with children under 6 years of age, to enroll in and 
successfully complete a comprehensive set of program components which are 
designed to enhance their employability. For example. New Jersey's REACH pro- 
gram provides in-program support services such as child care vouchers, half-fare 
transportation vouchers, and training-related expenses up to $30 per week. Florida's 
Project Independence and Michigan's MOST Demonstration will each pay for cosme- 
tic dental services which are not covered by Medicaid but are determined to enhance 
a participant's employability. 

Notably, many demonstrations continue to provide supportive services for 
up to one year after job placement. Again, using t le REACH program as an exam- 
ple, post-program supportive services include the provision of child care vouchers for 
up to one year f post-welfare employment and the extension of Medicaid benefits 
for one or more years of post-welfare employment. 

We also observed that adequate and stable funding was viewed as allowing 
innovative planning without uncertainty and narrow resource constraints. Funding 
was usually a mixture of currently available federal funds for public assistance 
programs— such as WIN and the Food Stamp program— and for employment and train- 
ing services— such as JTPA and the Employment Service— together with special 
allocations of state funds by the state legislature. 

For example, in the Penns^-^'^ania Joint Jobs Initiative, the state has made 
more than $20 million available for a one-year demonstration evaluation of its Single 
Point of Contact (SPOC) pilot projects. In Michigan, a $1.2 million state Hard-to- 
Serve Program Initiative has been funded through state agency resources in the 
following amounts: $250,000 each from the Department of Social Services and the 
state Employment Security Commission; $500,000 from the Department of Labor 
(JTPA Title IIA monies); and $200,000 from the Department of Education (8% educa- 
tion funds). 

Finally, in most cases the demonstration and reform initiatives included 
specific performance standards. As discussed in the next chapter, most states had 
developed performance measures to reflect their priorities and had incorporated 
them into local service provider contracts. For the most part, these contracts were 
performance-based, and included payment provisions that were tied to the achieve- 
ment of specific benchmarks, such as job placements and post-program retention. 
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In Massachusetts, for example, quality placements was relied upon as the 
chief performance measure to manage the state's ET/Choices program* This mea- 
sure is incorporated into fixed-unit price contracts with local service providers and 
reflects the expectation that program participants will be placed in jobs that pay a 
defined minimally acceptable wage rate* Overall, our respondents uniformly 
believed that including performance standards in service contracts was critical to 
the success of their efforts* 

If we take these initiatives as a reasonable reflection of the types of pro- 
gram models required to serve the v/elfare population effectively, then they would 
suggest that simply changing JTPA's performance standards will not necessarily lead 
to large changes in the program's targeting priorities and program mix. It would 
appear that other, more basic changes in program design and targeting requirements 
would also be necessary to forge a coordinated and more comprehensive service 
delivery approach. But even if such changes were made, it would still remain to 
determine if performance standards constrained service provider decisions. Since 
most of the welfare demonstrations adopted performance standards similar to those 
used in JTPA, we can gain some insight into this by comparing the perspectives of 
3TPA and welfare agency officials. We turn to this issue in the next chapter. 
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^.d PERFORMANCE STANDARDS, TARGETING, AND PRCXIRAM MIX 

On the basis of the information presented in the previous chapter, it is clear 
that states are devoting substantial levels of resources to programs and other initia- 
tives designed to improve services to AFDC recipients. Model program designs 
emphasizing interagency coordination, support services and remedial education, as 
well as case management characterize an increasing share of the program landscape 
in both the JTPA and the Welfare system. Yet within this, there remains a concern 
that performance standards constrain the choices service providers face in determin- 
ing the level and mix of programs to offer AFDC recipients. 

These concerns are being increasingly voiced, at least in the regular, ongo- 
ing employment and training program, heading many policymakers to conclude that 
the performance standards need to be more carefully managed and modified to 
better stimulate service delivery to AFDC recipients, and especially the hard-to- 
employ. This is particularly evident in the JTPA program, given its emphasis on 
performance management and the central role it plays in providing employment and 
training services to the disadvantaged population. The Senate Committee on Labor 
and Human Resources probably best captured this sentiment in its report accompany- 
ing the proposed Jobs for Employable Dependent Individual Act: 

The Committee conducted hearings on January 21, February 3, February ^, 
and March 6, 1987 to receive testimony on both the effectiveness of the Job 
Training Partnership Act in serving long-term welfare recipients, and the 
specific provisions and ramifications of S.!)i^. Witnesses commented on the 
extent and nature of long-term welfare dependency, particularly with 
regard to recipients of Aid to Families with Dependent Children (AFDC), 
and on those aspects of JTPA*s program design which contribute to "cream- 
ing." This practice has serious consequences for long-term welfare recipi- 
ents who seek employment. Creaming is the practice of preferential 
enrollment of those among the eligib!,, population who require the least 
assistance to prepare for a job. The present JTPA performance standards 
encourage creaming by rewarding the SDAs with the lowest training and 
placement costs per individual. As a result, individuals needing more exten- 
sive or costly training are neglected or underserved. 

The implication of this is that if performance standards are modified or, in 
an extreme case, eliminated, service delivery to long-term AFDC recipients would 
be substantially improved. Disincentives to serve this subgroup of the population 
would be eliminated, tar^^ting would be redirected, and program services would be 



Committee on Labor and Human Resources, U.S. Senate, Jobs for Employ- 
able Dependent Individuals Act: Report Together with Additional Views , Report 100- 
20, 100th Congress, Washington, DC, March 20, 1987. 
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modified to accommodate the needs of a relatively different client population. The 
JTPA program essentially wo^^ld voluntarily shift its focus to a more narrow subset 
of the eligible population and make those program design changes necessary to 
effectively service it. 

Unfortunately, there is little evidence that this outcome would occur. 
While studies of the JTPA program have shown some evidence that service providers 
engage ir activities that promote selective enrollment, it is not clear that they are 
motivated chiefly by the desire to achieve performance targets. There are other 
factors that rray also be shaping the types of targeting and program investment 
decisions made by service providers. 

As discussed in Chapter 1, one such factor is JTPA's decentralized gover- 
nance structure and broad focus on the economically disadvantaged. By design, the 
JTPA program encourages states and localities to make program and targeting 
decisions that best reflect their own perception of needs and priorities. These priori- 
ties may not always be fully consistent with existing or new national priorities, but 
they are consistent with the the structure of the program and its emphasis on local 
discretion. The outcomes of state and local decisionmaking may thus be viewed as 
the aggregate effect of a program that encourages diversity in state and local 
choices over whom to serve and .^hat programs to offer. Absent specific targeting 
requirements, it is difficult to argue that some groups, such as the hard-to-employ, 
are being excluded from receiving JTPA services that they otherwise should have 
received. 

Further shaping the outcome of JTPA targeting and program mix decisions 
is th'e requirement in Section l*l(a) that "each job training plan shall provide 
employment and training opportunities to those who can benefit from and who are 
most in need of such opportunities." Because the interpretation of this requirement 
is also left to local discretion, its application can also be expected to result in a wide 
range of target group and program mix selections. Moreover, it can be chow i that 
certain interpretations of those who can benefit from and are most in need of ser- 
vices are inconsistent with popular perceptions of the characteristics of the hard-to- 
employ. 

For example, one way to characterize the hard-to-employ is in terms of 
labor force status. Since the problem of disadvantagedness tends to be related more 
to a lack of labor force participation than low earnings o*' high unemployment, it is 
reasonable to say that the hard-to-employ are concentrated among those who report 
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being out of the labor force. Similarly, since the majority of public assistance 
resources are consumed by a relatively small proportion of recipients who experience 
long, continuous spells and have weak attachments to the labor market, it is also 
reasonable to assume that among the AFDC population, the hard-to-employ are, 
more often than not, out of the labor force. V/hile these groups are certainly -^mong 
those who are most in need of services, their weak attachment to the labor force can 
be used as the basis to argue that they cannot readily benefit from program partici- 
pation. 

Since program enrollment represents one vehicle to secure employment, 
weak labor force participation suggests little interest in seeking out services and 
hence, a limited ability to benefit from employment and training programs. It fol- 
lows then, that the unemployed might best fit the meaning of "benefit from and most 
in need of" services because they are among the eligible population (or subgroups 
thereof) and, by virtue of their job search behavior, have a relatively high level of 
interest in progrann participation. To the extent that service providers adopt this 
perspective, their actions can be interpreted as logical and consistent with the 
discretion offered by JTPA, though not necessarily with the objectives of serving the 
hard-to-employ. 

In addition to the eligibility requirements of JTPA, other factors may also 
shape the targeting and program mix decisions made by service providers. These 
include institutional impediments to coordination between JTFA and welfare, limita- 
tions on the use of funds for support services and stipends, decreasing levels of 
funding, and limited resources for remedial education. Taken together, these factors 
may well inhibit the ability of service providers to target resources on AFDC recipi- 
ents and especially those w!io are hard to employ. 

When considering the role that performance standards play in shaping 
service pro\7^der decisions, it is thus important to place *hem in the context of other 



For a description of the labor force status of the disadvantaged see: Glen 
Schneider et al.. An Assessment of Funding Allocation Under the Job Training Pait- 
nership Act , Abt Associates Inc., Cambridge, MA, 1986. 
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See, for example, Mary Jo Bane and David T. EUwood, Slipping Into and 
Out of Poverty; The Dynamics of Spells , Harvard University, Cambridge, MA, 
August 1985; Mary Jo Bane and David T. EUwood, The Dynamics of Dependence: The 
Routes to Self-Sufficiency , Urban Systems Research and Engineering, Inc., Cam- 
bridge, MA, June 1983; and David T. EUwood, Targeting "Wou Id-Be" Long-Term 
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potentially important influencing factors. While JTPA's performance standards may 
well have some effect on local decisionmaking, they may not be the mooc important 
influencing variable. It could well be that absent other constraints, performance 
standards would have only a limited effect on program targeting and mix decisions. 
Thus, the key question is not so much whether performance standards constrain local 
discretion, but, relative to other factors, whether performance standards have an 
overwhelming influence. Only in this way is it possible to determine if changes in 
JTPA*s performance management system will redirect its orientation to AFDC 
recipients, and particularly those that are hard to employ. 

The telephone interviews we conducted with state-level representatives 
from JTPA and Welfare offices were designed to provide some insight into this 
question. In addition to asking respondents about programs and policies to encourage 
services to AFDC recipients, the interviews were also designed to document the 
current status of performance standards management, to identify respondent percep- 
tions of key factors inhibiting services to AFDC recipients, and to examine those 
steps which states believe are most likely to improve the level and quality of ser- 
vices to the welfare population. In the remainder of this chapter we report the 
results obtained from these interviews. 

1 The Cu r rent Status of Performance Standards Management 

Unlike programs authorized under JTPA, those administered by state 
departments of public welfare do not operate with federally required performance 
measures; each state relies on its own discretion to determine whether to use per- 
formance measures or not, and how those measures should be translated into numeric 
standards and applied for management purposes. Nonetheless, our results indicated 
that seven of the nine welfare departments included in the telephone survey rely 
upon performance measures to guide their programs. Like JTPA, these measures are 
based primarily upon labor market outcomes, such as placement in unsubsidized 
employment, average wage at placement, cost per placement, and employment 
stability 30 and 90 days after the termination from a program. Thus, we can say that 
employment and training programs serving a large share of AFDC recipients in the 
U.S. operate with performance measures that stress a closely related set of immedi- 
ate labor market outcomes. 

It is also true that when setting numeric standards, efforts are made to 
account for certain factors that affect performance achievement. The manner in 
which this is done, and more generally, how performance standards are managed. 
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differs markedly between JTPA and programs offered by state departments of public 
welfare. For the most part, welfare departments do not set explicit statewide 
rtandards and then apply them to contractors. Rather, welfare departments tend to 
manage performance on a vendor-by-vendor basis, relying upon fixed-unit price 
contracts as the vehicle to express performance expectations. While the nature of 
these contracts varies across and within states, they all operate on the basis of 
reimbursement for the achievement of agreed upon performance outcomes, such as a 
job placement. The exact mix and value of these reimbursement outcomes (or 
benchmarks, as they are often called) are determined through contract negotiations 
that, in most cases, take account of the type of client served and local economic 
conditions. 

In contrast, the JTPA program manages performance in a much more elab- 
orate and sophisticated manner. All of the nine JTPA offices we interviewed rely 
upon the Department of Labor's statistical adjustment model to set the adult v/elfare 
entered employment rate, while six of the nine also have policies for further negoti- 
ated adjustments. These policies adhere very closely to the procedures and methods 
recommended by the Department of Labor. In general, they require an SDA to 
explain why the adjustment is needed, the level of adjustment requested, a justifica- 
tion for the additional adjustment, and the method used to derive the adjustment. 
This then forms the basis of negotiation between the SDA and the state. 

The use of the Department's adjustment model is viewed by states as the 
most objective means for removing disincentives associated with serving hard-to- 
employ participants. In explaining why his state chooses to use the DOL model to set 
the adult welfare performance standard, one JTPA respondent expressed a common 
viewpoint: 

We have to have a reasonably fair way for setting SDA expectations of 
performance. The DOL model provides a way for the state to treat all the 
SDAs the same. We have to have performance standards that will stand up 
if they are challenged by DOL; so we decided to take the path of least 
resistance. 

However, most states we interviewed believe that the adjustment models 
are limited by the omission of several important factors that affect employability, 
such as functional literacy, motivation, and degree of dependence on public assis- 
tance. As a result, six of the nine states introduced policies to support adjustments 
beyond those provided by the Department's models. These policies follow the guide- 
lines outlined by the Department of Labor and are intended to further limit any 
constraints performance standards may impose on targeting and program mix deci- 
sions. 
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In addition to adjustments beyond the model, states can also rely upon their 
six percent grants to provide incentives to serve hard-to-employ participants, such as 
AFDC recipients. Of the nine 3TPA offices we interviev^ed, five had earmarked a 
portion of their six percent grants for this purpose, and of the five, four had provi- 
sions targeted on AFDC recipients. 

The portion of six percent grants dedicated to the hard-to-employ varied 
across these states as did the criteria for determining whether an SDA would be 
eligible for receiving the additional funds. In a few of the states, when calculating 
incentive awards, hard-to-serve participants are omitted from the base calculations 
of performance achievement to hold SDAs harmless from any effects these partici- 
pants may have on program outcomes. In other states, SDAs must meet their base 
performance standards requirement and also show evidence that they served hard-to- 
employ participants at a greater rate than they would normally be expected to. 
Finally, the amount of six percent funds devoted to the hard-to-employ varied from 
as little as 26 percent to as much as all of the six percent grant. These types of 
variation in the scope and level of six percent incentive funds devoted to the hard- 
to-employ clearly reflects the discretion provided to statt:> by the 3TPA legislation. 

Difference.*^ in the mar ^err.ent of performance standards between 3TPA 
and departments of public welfare were evident in ways other than just technical 
considerations. In comparison to 3TPA, the management of performance standards 
by welfare officials was viewed as an important, but nonetheless regular, ongoing 
responsibility along with other key management tasks. In 3TPA, however, perfor- 
mance standards represented a much more significant policy position and consumed 
considerably more resources in the form of staff specialization and policy direc- 
tives. To a large extent, our 3TPA interviews indicated a performance standards 
culture, emanating partly from the Department of Labor, and taking on a life of its 
own at the state level. This difference is quite important in understanding the 
differences in perspective between 3TPA and welfare with respect to the constrain- 
ing influence of performance standards, 

^.2 Serving AFDC Recipients; The Role of Performance Standard:; 

If performance standards are hypothesized to substantially constrain service 
delivery to AFDC recipients, and those that are hard to employ, then the qualitative 
results obtained from our telephone interviews would suggest only limited support for 
the hypothesis. Among all eighteen 3TPA and welfare respondents, six, or only one 
third, cited performance standards as among the most important factors inhibiting 
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state and service provider (SDA) decisions regarding the provision of services to this 
population group. The most important factors cited included limited funds and 
perceptions regarding the role of employment and training programs in limiting 
welfare dependency. 

In some instances, the is. le of perceptions took the form of a limited social 
consensus on how to address welfare dependency. One of our respondents indicated, 
for example, that: 

We don't really k^r>w what we as a society want to do with welfare recipi- 
ents because we aon't want them to stay on the rolls but we don't really 
want to do anything substantial to improve their lives. 

In other cases, respondents commented on the influence of the public's 
perception of the average welfare recipient. As one respondent put it: 

Even if there was unlimited funding to provide E/T services to welfare 
recipients, our state's decisions about how much to invest in a welfare 
recipient would still be influenced by the average taxpayer's view that you 
should get these deadbeats off the welfare rolls." 

These varying and often conflicting views of how to reduce welfare dependency 
created some element of confusion and tentativeness at the state level, and fre- 
quently were interpreted as having to implement "quick-fix" program solutions. 

When respondent views of the role of performance standards are reported 
separately for 3TPA and welfare offices, an expected difference emerges. While 
standards were mentioned only once by welfare respondents, they were cited by five 
of the nine JTPA respondents. For the most part, JTPA officials saw performance 
standards as constraining targeting decisions largely because of the risks they pre- 
sented to SDAs and their service providers. In some cases, thece risks were 
explained in terms of a potential loss of six percent funds, while in other instances 
respondents cited the political necessity of achieving the standards, :jd the priority 
placed upon them by members of Private Industry Councils. Finally, the incorpora- 
tion of performance expectations in fixed-unit price contracts was also cited as 
limiting vendor ability to enroll AFDC recipients, including the hard-to-employ. 
From the perspective of a vendor, enrolling the hard-to-employ placed reimburse- 
ment for expected performance at risk as well. 

The fact that JTPA respondents cited performance standards as a 
constraining factor much more often than their welfare counterparts is interesting 
and potentially informative. Since welfare departments, like JTPA, rely heavily 
upon performance expectations, the limited importance they placed on standards 
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may reflect the fact that they do not operate under an elaborate, legislatively man- 
dated performance standards system. It became clear during our interviews that 
JTPA's emphasis on performance achievement had created a relatively complex and 
high priority activity at the state level l.iat was carried out within a sophisticated 
and technical structure. We found an emotionally charged audience of JTPA respon- 
dents, some strongly for and others against performance standards, as well as a man- 
agement tool that seemed to be set apart from others routine*, used by program 
administrators. Welfare departments, in contrast, relied upon performance stan- 
dards, but approached them in much the same way they did other management con- 
cerns. While performance achievement was indeed considered important, it did not 
occupy the same status as among our JTPA respondents. Performance standards 
management was simply not as difficult or problematic in the highly targeted wel- 
fare programs we observed as in the more brocvdly defined JTPA program. This may 
well imply that a key problem with JTPA's performance standards is that they have 
become too prominent and too complex in the minds and management systems of 
program administrators, overriding other key management concerns and the primary 
importance of focusing on individual entpi'^vability development. It may also imply 
that it is possible to manage with performance standards in programs that are highly 
targeted on the hard-to-employ. 

Although JTPA respondents cited performance standards as inhibiting 
services to AFDC recipients more often than welfare respondents, it still repre- 
sented only a little mere than half ol all potential responses. This mixed message 
was reinforced by the responses to our question regarding the best ways for states 
and service providers (SDAs) to encourage or improve services to AFDC recipients, 
including those that are hard-to-employ. 

As before, across all respondents, the single most frequently cited responses 
included increased funding and better coordination between JTPA and welfare pre 
grams, followed by better incentives and improved management of performance 
standards. Importantly, the two most important responses given by JTPA respon- 
dents included improved coordination and better incentives. Each is addressed in 
turn below. 

Inter-agency coordination was a major theme running throughout our inter- 
views. Regardless of which respondent, it was viewed as a critical step for improv- 
ing service delivery to AFDC recipients. For example. New Jersey*s demonstration 
program— REACH— includes specific policies to promote the delivery of remediation 
services to welfare recipients. Since the REACH budget includes limited resources 
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for such services, the state welfare department negotiated an interagency arrange- 
ment with the state Department of Education. As a result, the education depart- 
ment has targeted some of its adult basic education monies (from both state and 
federal non-3TPA sources) for REACH clients. The state department of education 
will also participate in the county-level decision process on how these funds should 
be targeted to REACH clients. 

At both the state and local level, competition for clients, turf battles, 
incongruity in target group focus, and differing expectations regarding the role of 
employment and training services were viewe as limiting the coordination of two 
programs that otherwise shared much in common. For example, a 3TPA official 
observed that: 

3TPA dollars could be sfetched further and more people could be served by 
an SDA it there .vas improved coordination with local welfare departments 
to provide s^-pportive services and with local education agencies to provide 
remediation services. 

Improving incentives was an issue cited more often by 3TPA than welfare 
respondents. This reflected two typeu of concerns among the 3TPA respondents. 
The first was that despite policies and procedures to support adjustments beyond the 
Deportment of Labor's model, few if any SDAs had requested such adjustments. One 
3TPA respondent observed that: 

I don't see enough current use of the adjustment procedures to b .ileve that 
more flexibility would make any difference. 

Our respondents attributed this to three factors, including the complexity 
and data requirements of the additional adjustment procedures, the ease with which 
SDAs could achieve their performance targets without additional adjustments, and 
their concern over not looking bad! Apparently, many state officials believe that as 
long as SDAs can achieve their performance expectation, they have little incentive 
to go beyond it. 

The second concern over incentives was financial. In several of our inter- 
views, respondents cited the limited effects that six percent policies were having on 
encouraging services to the hard-to-employ, including AFDC recipients. While most 
agreed that the structure and intent of the six percent grant had merit, 'ts relatively 
small size was having only a limited effect on the behavior of SDAs. For example, a 
3TPA official observed that: 

Performance standards don't have anywhere near the importance that many 
people claim they do. For example, most of our SDAs met the youth per- 
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formance standard and knocked them all to hell. I: PY1986, we could have 
doubled the cost per and all our SDAs still would have met these stan- 
dards. And, due to the structure of the incentive formulae, there is no 
incentive to an SDA to exceed the performance standard by more than 10 
percent; yet most of our SDAs exceeded them by 100 percent. 

On t^e basis of our interviews, it is not clear that a strong case can be 
made for performance standards being a major factor in constraining service pro- 
vider decisions regarding whoni to serve and in what programs. This is certainly the 
case across both JTPA and v/elfare programs and within piugrams administered 
directly by welfare agencies. While our JTPA respondents did cite performance 
standards most often, it represented just over one-half of all responses, in compari- 
son to an overwhelming sentiment. Moreover, when we examined how states would 
improve services to AFDC recipients, modifying the performance standards system 
was mentioned only after improving coordination and incentives. It would thus 
appear that while changes to the performance standards system may be necessary, 
they are not a sufficient condition for enhancing the level and quality of services to 
the welf 3 population. 

4.3 Improving Services to Welfare Recipients 

The road to improving services to AFDC recipients appears to lie in the 
direction of coordination and support services. When respondents were asked to cite 
those changes that are most important to improving services to AFDC recipients, 
these two factors were cited most often, regardless of whether the respondents 
represented JTPA or the department of public welfare. 

Welfare recipients* multiple barriers to emoloyment is what drives the call 
for increased interagency coordination because it is widely acknowledged that no 
single program or .'3enry c^r, be equipped to handle tb multiple and varied needs of 
such clients, particularly tne hard-to-employ. Several respondents noted thf. linkage 
between effective interagency coordination and programmatic success as evidenced 
by the following representative comment: 

Our new state-wide welfare reform initiative has brought together welfare 
offices, JTPA, schools, and human services agencies which have never 
'A^orked together previously. To make the initiative a success, these agen- 
cies mu* ^t work together. 

A second rationale that led respondents to emphasize coordination issues is 
the political envirr^nment of budget cutbacks, particulaiiy for social welfare pro- 
grams. Respondents strongly believe that more cooperation between JTPA, the state 
welfare department, and other related agencies would result in better matclung oi 
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welfare recipients* service needs with the existing employment and training and 
educational program slots as well as with currently available supportive services. To 
require each service delivery system to provide a fully comprehensive array of 
services may neither be economically efficient nor necessarily guarantee the appro- 
priate level and mix of services. To illustrate this point, a recent state RFP 
announced a new competition for demonstration projects to serve hard-to-serve 
individuals, A key goal of these projects is io: 

..•develop an integrated system of service delivery which is transparent and 
therefore easily accessible to individual users and which makes maximum 
use of existing resources to identify and meet their needs.... It is not the 
intent of this RFP to set up a separate delivery structure for serving hard- 
to-serve individuals, but rather to expand opportunities through existing 
programs. Therefore, the proposal should specify how the provision of 
services will be integrated into existing programs to make maximum use of 
existing resources available in this area. 

As a final point, we observed that several of our respondents talked about 
the need for both formal and informal approaches to interagency coordination. 
Sources of formal coordination were statutory provisions of new welfare reform 
initiatives, the Governor's Coordination Plan, mission statements, RFPs, and other 
policy directives. Sources of informal coordination mainly consisted of meetings, 
talking, and planning with one^ counterparts in other agencies. The following state- 
ment in a 1987 annual report to the Florida state legislature illustrates the necessity 
for such informal interactions: 

Formal organizational agreements and cooperative working agreements will 
provide staff with the framework of responsibilities required to implement 
Project Independence. However, it is often Zhe informal relationships 
formed at the local level which make interagency coordination and cooper- 
ation effective. Developing informal interagency relationships will require 
meeting staff from agencies who provide services and resources needed by 
Project Independence participants to achieve self-sufficiency and indepen- 
dence from welfare. 

Both Welfare and JTPA officials also see a critical need for supportive 
services which are more comprehensive, richer, and longer than those which are 
typically available. Several of them observed that die provision of such enhanced 
supportive services was an absolutely necessary ingredii^nt to motivate welfare 
recipients to enroll and then to successfully complete a series of program compo- 
nents which will lead to economic self-sufficiency. Without such services during 



Florida Department of Health and Rehabilit^.tion :)crvices, Project Inde- 
pendence Report , 'Tallahassee, FL, August 1987. 
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program participation, the client has a higher likelihood of becoming discouraged and 
dropping out, due, for example, to an inability to manage both family and program 
participation responsibilities. 

The extension of such services, even after a recipient obtained nonsubsid- 
ized employment, was also stressed as a key fa: or in preventing recidivism. To this 
end, several respondents reported that their states are now extending health benefits 
for up to one year post-employment since many welfare recipients are unable to 
qualify for jobs that provide important fringe benefits such as medical insurance. 
Finally, respondents are also aware that the availability of supportive services often 
motiv^ates program operators to make longer-term investments in welfare recipi- 
ents. One JTPA respondent spoke directly to the dual motivating role that suppor- 
tive services can play: 

The success rate of long-term employment and training programs has been 
weakened by a lack of consistency in support services. The longer the 
program goes, the more chance that the supportive services will break down 
and as a result, the participant will drop out. As a result, program opera- 
tors donH want to take the chance of committing money to hard-to-serve 
clients or targeted programs which have a lower chance of success. 

Although several of our respondents also expressed a concern that perfor- 
mance standards constrain targeting and program mix decisions, they were clear that 
the standards were not as important as other factors. The respondents did, however, 
offer suggestions on how to improve JTPA^s performance m'^nagement system. First, 
in a few cases respondents cited a need for a more general outcome measure, such as 
a positive termination rate, to assess programs serving welfare recipients. They 
believed that broadening the focus of performance expectations would enable service 
providers to concentrate on a wider re '',e of employability development activities 
such as remedial education. 

Second, severa. respondents believed that the adjustment models could be 
improved through the inclusion of additional variables, such as duration on public 
assistance, to better control for key factors affecting the achievement ot perfor- 
mance outcomes. Such changes, however, would also have to be accompanied by 
demystifying and simplifying the performance standards procers in order for SDAs to 
both better und 'stand it and, when necessary, request additional adjustments. 

But perhaps most clear is that modifications to the performance standards 
system alone will not necessarily have a marked effect on targeting and program 
decisions. Other factor . such as coordination and support services, would have to be 
addressed as well. 
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Exhibit 

Use of Performance Standards by Welfare Departments 



Question 

Do you rely on specific performance 
measures to manage employment and 
training services? 



Number of 
Respondents 
Answering 
"Yes" 



Number of 
Respondents 
Answering 
"No" 



Measures that are relied on: 

Output Measures 

Job placements 
Wages at placement 
Cost per placement 
Post-termination measures 

Input Measures 

Participation 
Target group shares 
Cost per participant 

Do you set state-wide performance standards and then 
apply them to contractors, or are the standards 
set on a contract-by-contract basis? 

State-wide standards 
Contract-specific standards 
Both 



Number of 
Respondents 
Answering 
"Yes" 
(N = 7) 



7 
6 
3 
6 



I 
0 



1 
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Exhibit ^.2 

Use of Performance Standards by Welfare Departments 



Question 

Use of adjustment factors in setting 
performance standards or benchmarks: 

Client characteristics 
Local economic conditions 
Size of local caseload 

Use of specific incentives to encourage 
good performance 



Number of 
Respondents 
/ iswering 
"Yes'' 



Number of 
Respondents 
Answering 
"No" 



7 
6 
1 



2 
3 
8 
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Exhibit h.} 

Procedures Used to Adjust 3TPA Performance Standards 



Methods Used to Set 

Adult/Welfare Performance Standards 

Use of DOL adjustment models for 
adult welfare EER 

Use of the national standard 

Use of the adult welfare EER ratio 

Use of further state-SDA negotiated 
adjustments to the four adult 
i>erformance standards 

Adult Ec.R 
Aduit average v\ge 
Adult cost/EE 
Adult welfare EER 

Use of additional performance measures 
for adult welfare programs 



Number of 
Respondents 
Answering 
"Yes" 



Number of 
Respondents 
Answering 
"No" 



0 
0 



6 
6 
6 
6 



0 
0 



3 
3 
3 
3 
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Any 

Portion 
to Hard- 
to-Serve? 



No 
No 



Exhibit 

Use of 6% Funds As Incentives to Encourage ^yervices to Hard-to-Serve Groups 



Percent of 6^ Funds 



Definition of Hard-to-Serve or 
Target Grou p 



H ow to Become Eligible 



Constraints on Funds Used 



Massa- 
chu- 
setts 



Yes 



Mich- 
igan 



ON 
ON 



Yes 



In PY86, $500K is targeted 
towards hard-to-serve from 
$t.2M 6% funding (about 40j( of 
1he tota!) 



SDAs can choose to use aU or 
part of 6% funds awarded for 
PY86 performance to provide 
services to hard-to-serve 
cl ients. 

8W ($6.85M) of PY86 6^ funds 
are available for incentive 
grants (S7.7M in PY86) 

Participants identified as 6% 
hard-to-serve were not 
included in PY86 calculations 
for purposes of awarding 6% 
incentive grants (see Note 1). 



High school dropouts 
Minorities 



An individual who is: 

-Economically disadvantaged 
-Unemployed +2 years or never 
*%n employed 

need of functional liter- 
acy or employabi I ity charac- 
teristics or supportive 
services 

-Whose Employabi llty Develop- 
ment Plan indicates that the 
duration or cost of training 
leading to employment exceeds 
that received by the average 
program participant 



SOAs must achieve both a high 
rate of service and excellent 
Outcomes as measured by high 
placement rates and placement 
wages. Exact amounts at award 
depend on the percent by which 
the SDA exceeds established 
benchmarks. 

N/A 



N/A 



70% training limitation 

No mention of administrative 
cost limits 



Note 1: To be eligible for exclusion from incentive award calculations, participants must meet the following definition of hard-to-serve; 
a) An individual mus^ be economically disadvantaged and either meet 2 of the following: 
-School dropout -Offender 
-Teen parent -Substance abuser 

Or, one of the above and at least one of the folloi/ing: 
-Youth 

-AFOC recipient 

-Handicspped ^ .-^ 

0 b 




StaK« 

New 
Jersey 



Yes 



New 
York 



Yes 



Exhibit i^A (continued) 
Use of 6% Funds As Incentives to Encourage Services to Hard-to-Serve Groups 



Any 

Port I on 
0 Hard- 

to-Serve? 'Percent of 6% Funds 



Approximately 26% of total 6% 
pool is targeted towards hard- 
to-serve c snts 

1/3 of eO% of total pool is 
available for SDAs that exceed 
all performance standards; 
however, SDA can request a 
waiver f r ^ exceeding the 
Welfare performance standards 
if it is serving REACH parti 
cipants by double the WIN 
Title IIA requirement. 



$3«t2M was awarded for 6^ 
hard-to-serve in PY86 



Definition of Hard-to-Serve or 
Target Group 

Hard-to-serve groups may 
nc I L'de : 

-i^EACH participants with less 
than a 6th grade reading 
level 

-AFDC participants with 2 

preschool chi Idren 
-Severely handicapped 
-Non English-speaking 
-Ex-of fenc'frs 

-School dropouts under age 18 
-Displaced homemakers over age 
40 

-Substance abusers 

-Homeless 

-Dropouts 

-Other group which the SDA 
demonstrates are unusually 
hard to serve 

A. Priority Groups 
-Minority youth and adults 
-Single head of household 

B. Secondary Groups 
-Handicapped 
-School dropo'jts 
-Limited ESL proficiency 
-Offenders 

-Alcoholics and addicts 
-Veterans 

-AFOC recipients, including 
voluntary programs for women 
with children under 6 



How to Become El iqible 

SDAs must exceed minimum of 3 
out of 4 adult performance 
standards and 2 out of 3 youth 
performance standards. One of 
the adult perfoma'^ce stan- 
dards must be cost per. 

SDAs must submit documentation 
that they served seiect hard- 
to-serve Individuals over and 
above the number that the SDA 
would normally be coected to 
serve. 



Must qua I if y for a PY86 
performance award 

Must exceed 1 or both of the 
Adult Welfare performance 
standard and Youth EER perfor- 
mance standard. 

Must submit an Implementation 
plan for the incentive award. 



Constraints on Funds Used 

70/30 cost spl it 

No 151 limitation on adminis- 
trative costs 



Funds mi'St supplement and not 
supplant services available 
currently. 
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UC 



state 



Ohio 



Yes 



Exhibit (continued) 
Use of 6% Funds As Incentives to Encourage Services to Hard-to-Serve Groups 



Any 

Portion 
to Hard - 

to-Serve? Percent of 6^ Fufids 



B0% is available for Incentive 
Grants. SDAs who receive 
grants must spend 25% of funds 
on hard-to-serve clients. 



Definition of Hard-to-Serve or 
Target Groun 

A. State Categories 
-Women 
•Minorities 

-Older workers (40-70 years) 

-Handicapped 

-Veterans 

• Dropouts 

-Female household heads 
-18-25 year old black males 

B. Local Catei7 ^r!es 

-SDAs may also define 2 local 
hard-to-serv<» categories 



How to Become Eligible 

SDA must meet all / perfor- 
mance standards and exceed at 
least one performance standard 



Constraints on Funds Used 



N/A 



Penn- 
sylvan- 
ia 



No 



Texas 



No 



ON 
00 
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Exhibit ^fj 

What Are the f - 3t Ways for States and Service P rnviHpr<; t.^ 
fcncoiiraRe/Improve :)ervic e$ to Hard-to-i^.erve Welfar^"RiEHents? 



Pol tcy Tool 

Better management of performance standards 
Better management of fixed-unit price contracts 
Increased funding 

Increased or Improved, technical assistance 
FinancJal Incentives 

Better coordination with welfare/JTPA departments 
BetteiVmoro practical ways of Identifying hard-to-serve 
Other 



Number o f R^_^^onses 
JTPA 



3 
0 
3 
1 

5 
5 
2 
5 



Welfare 

3 
0 
5 
2 
2 
3 
0 
6 



Total 

6 
0 
8 
3 
7 
8 
2 
11 



L^fen'tT"""^ ""^'^ mentioned as a key way to encourage services to AFDC 

What are the ways in which performance standards can be improved? 



Policy Tool 

Helpfui to change the mix of standards? 

Helpful to provide better Incentives? 

Helpful to achieve better local management/changes in 

fixed-unit price (Witracts? 
Helpful to develop-more flexible adjustment procedures? 



Number of Responses 
JTPA (N=3) Welfare (N=3) 



Yes 


No 


OK 


Yes No 


2 


0 


1 


3 0 


2 


1 


0 


2 0 


2 


0 


1 


1 1 


3 


0 


0 


1 0 



DK 

0 
1 
1 



A^D^recip^^ ^ mentioned as a key way to encourage services to 

What are the rays m mth performance standards can be improved? 



Policy Tool 

Helpful to change the mix bf standar Is? 

Helpful to provide better incentives? 

Helpful to achieve better local management/changes in 

fixed-unit price contracts? 
Helpful to develbp mor* flexible adjustment procedures? 



Number of Responses 
JTPA (N=6) Welfare (N=6) 

"ok 



Yes No OK Yes No 



2 
2 
1 



0 
0 
3 



3 
« 

2 



2 1 
2 1 



Not asked 
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Exhibit 

Mos<: Important Factors which Inhibit State and SPA (Service Provider) 



Decisions to Serve Hard-to-Serve Welfare Recipients 



Factor 



Cost I imitations 

Limited funds 

Performance standards 

Limited capability/knowledge 

Political pressure 

Support services limitations 

Limited incentives 

Difficulty in identifying this group 

Pv-rceptions 

Coordination 

Other 



Number of Responses 
JTPA Welfare 



Total 



0 
6 
2 

1 

% 
I 

2 
0 
0 
7 
1 

0 



t 

8 
7 
2 
1 
4 
0 
0 
9 
3 
2 



If performance standards were mentioned as a chi^^f inhibiting factor: 

Do performance standards play a key role in inhibiting policy objectives concerning 
provision of services to the hard-to-employ? 

Number of Responses 
JTPA (N=5) Welfare (N=2) 



Pol icy Tool Yes No Yes No 

Serving the hard- 'o-employ 4 1 2 0 

Offering longer-term services 3 2 2 0 

Creatinp an effective natch between client need and service mix 4 1 2 0 



If performance standards v -^r > not mentioned as a chief inhibiting factor: 

Do performance standards play a key role in inhibiting policy objectives concerning 
provision of services to the hard-to-employ? 

Nmnber of RcSponi^es 
JTPA (N=4) Welfare (N=7) 



Pol icy Tool Yes No Yes No 

Serving the ht^rd-to-employ 3 1 5 2 

Offering longer-term services 2 2 4 3 

Creating an effective match between client need and service mix 13 3 4 
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Exhibit iiJ 



Changes in Policy Tools Which Are Viewed As Most Important in Improving the Level 
and Quality of Services to AFDC Recipients 



Selected As 
Most Important 



Selected As 
First Priority 



Selected As 
Second Priority 





Pol Icy Tool JTPA 


Wei fare 


JTPA 


Wei fare 


JTPA 


Welfara 




G 


Improving coordination with welfare departments 


6 


5 


3 


2 


3 


3 


</> 

o 


Better identification and selection of the hard-to-employ 


2 


3 


0 


0 


1 


1 


a 
•1 

U 
t* 


Inproving/augmenting technical assistance 


3 


4 


0 


1 


1 


1 


:> 
«* 

•1 


Chdnginy or improving incentives 


3 


4 


1 


2 


0 


1 


IS 
« 

o 


Augmenting availability of support services 


5 


5 


4 


2 


2 


2 


»•» 
»s 

n 
a 


Changing the mix of psrformance stan^iards 


2 


3 


0 


0 


0 


0 


•o 

CO 


Changing the adjustment procedures to be more flexible 


4 


4 


1 


1 


0 


0 


i >— 


Better management of fixed-uni?- price contracts 


1 


1 


0 


0 


0 


0 
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